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ABSTRACT 



This thesis presents an evaluation of three relatively simple reli- 
ability growth models for which accuracy, precision, and robustness per- 
formance were examined over a wide variety of true underlying reliability 
growth patterns. A continuous cumulative failure rate model, a continu- 
ous instantaneous failure rate model, and a discrete reliability model, 
each of which employ ordinary regression methods, were evaluated using 
standard computer Monte Carlo simulation techniques. Simple, straight- 
forward statistical measures of performance are exhibited in graphical 
and tabular form. All the models displayed some degree of difficulty in 
tracking particular types or portions of anomalous reliability growth 
patterns. The cumulative model displayed this difficulty the least and 
exhibited good variability (precision) performance providing confidence 
in its use. The instantaneous model, while displaying generally good ac- 
curacy, exhibited poor variability performance. Except for a couple of 
anomalous situations, the discrete model showed good accuracy and vari- 
ability performance. Forecasting performance of all the models proved 
to be worse than their capability to determine current reliability 
status . 
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INTRODUCTION 



New weapons systems components characteristically display very low 
initial reliability as they begin progress through early development and 
testing. This low infant reliability of a system and its components is 
common to many commercial systems such as automobiles, aircraft, etc. 
as well as complex military systems. The reticence of contractors to 
divulge early test data that reveals this low infant reliability can 
handicap the project officer and contractor in exercising their manage- 
rial responsibility to bring the system in on time at the specified re- 
liability level. Because reliability growth is not tracked in early 
development stages, timely opportunities to either accelerate or inten- 
sify development by reallocating funds during the initial design/devel- 
opment phases of a system acquisition cycle are frequently lost. Conse- 
quently, managers can lose an opportunity to support their projects at 
a time of greatest need. 

There exi sts an abundance of reliability growth models designed to 
assist managers in charting the reliability status of a system and its 
components during any stage of development and testing. Unfortunately, 
the benefits, accuracies, and comparisons of these models are not always 
understood by management staff who could profit from their use. 

The primary purpose of this thesis was to evaluate three easily 
understood, relatively simple reliability growth models which would 
hopefully prove to be reasonably accurate over a broad spectrum of 
reliability growth situations. Conditioned upon this demonstrated 
versatility in a wide variety of circumstances, the ultimate goal is 
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for employment and utilization of these reliability growth models by 
management on a significantly larger scale with increased willingness 
to share reliability progress information between contractor and custom- 
er. The evaluation was intentionally not structured to ascertain the 
capabilities of the reliability growth models as high resolution tools 
in very restricted situations but rather as gross models for flagging 
significant deviations of a system's or component's reliability progress 
from a desired or expected reliability growth pattern. Hence, the 
models' ability to alert managers that consideration of corrective action 
or at least more intense scrutiny of reliability status is required was 
established as a goal of the evaluation. Perhaps the greatest potential 
windfall benefit of utilizing capable reliability growth models as a 
project management tool is a reduction in cost overruns that have plagued 
weapons systems development since the 1950's. A primary cause of weapons 
systems cost overruns as noted in reference 3 is trying to develop and 
produce the systems too fast. 

Since the underlying reliability progress pattern of a new system 
and its components is unknown to the project officer and contractor, 
these managers essentially need a "model for all seasons". In this vein 
the attributes desired in a reliability growth model are accuracy and 
robustness. Hence, a model (s) that can discern the true underlying re- 
liability progress pattern with some degree of fidelity (accuracy) with- 
out sacrificing the capacity to cope with a significantly diverse range 
of reliability progress patterns (robustness) is the tool managers seek. 

With these attributes in mind a continuous cumulative failure rate 
reliability growth model, a continuous instantaneous failure rate reli- 
ability growth model, and a discrete reliability growth model were chosen 
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for evaluation. Since each model is structured to measure a different 



parametric indicator of reliability progress, no direct comparisons 
between the models' performance were made. The models are presumptuous 
as the generally accepted title reliability growth models Implies. In 
order to evaluate the models' accuracy and robustness not only were they 
tested against "nice" underlying reliability progress paths, but also, 
they were confronted with underlying patterns which exhibited relia- 
bility stagnation and degradation prior to continued growth as well as 
patterns demonstrating no reliability progress. 

The three reliability growth models evaluated in this thesis are 
designed to be very general in the nature of the items for which they 
propose to chart reliability progress. The models can be employed for 
reliability tracking at the system level, sub-system level, and/or the 
individual component level during any phase of an acquisition cycle to 
include even retrofit programs. Hence, for the balance of this thesis 
"component" and "item" are used to reference an entire system, or a 
sub-system, or an individual component for which reliability progress 
is being modelled as appropriate. 

The evaluation technique utilized computer simulation which permits 
the underlying reliability progress patterns to be specified while 
observing the models' performance in both determining and forecasting 
the underlying progress patterns. The simulation technique and evalua- 
tion measures are described in the next chapter. 
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11. EVALUATION METHOD AND MEASURES OF EFFECTIVENESS 



A. EVALUATION METHOD 

Performance evaluation of both continuous failure rate reliability 
growth models and the discrete reliability growth model was accomplished 
using computer Monte Carlo simulation. Computer simulation permits the 
analyst to specify and control the underlying reliability path of growth 
By utilizing simulation data that is equivalent to data normally avail- 
able in a testing and development program the models' performance may 
be compared with the specified underlying reliability path. 

The continuous cumulative failure rate reliability growth model and 
the continuous instantaneous failure rate reliability growth model both 
utilize the same data from a testing and development program. Therefore 
these two models were evaluated simultaneously in a unified simulation 
program. The test program assumed for the continuous models is essen- 
tially a deterministic program in which testing of items in any given 
test phase can be accomplished simultaneously since the number of items 
to be tested within each phase is specified prior to conduct of the 
test phase. On the other hand the discrete reliability growth model is 
based on a sequential test structure in which the number of units tested 
cannot be specified beforehand; specifically, testing in each phase is 
continued until a specified number of failures are observed which in 
turn generate a significant development or production change that hope- 
fully Improves reliability. Consequently, the number of tests within a 
test phase are random. Because of this dissimilarity in test program 
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structure for the continuous and discrete models, a separate simulation 
program was run for evaluation of the discrete reliability growth model. 

B. MEASURES OF EFFECTIVENESS 

As indicated in the introduction, the primary goal of this evaluation 
was to demonstrate the capability of selected reliability growth models 
to estimate unknown, underlying reliability progress paths with suffi- 
cient accuracy and versatility. Thus, the capability of the models to 
satisfy these requirements is a question of their accuracy and precision; 
i.e., (1) how close do the models* reliability or failure rate estimates 
approximate the true underlying reliability or failure rate paths and 
(2) are the models consistent in their approximations? 

To answer these two questions regarding the models' capabilities the 
simulations were replicated one-hundred times for each underlying reli- 
ability progress path specified. The number of replications was actual- 
ly programmed as a "user specified" input variable and could have been 
chosen to be ten runs for one progress path and a thousand runs for 
another progress path. One-hundred replications for all progress paths 
was chosen as a sufficiently large number to provide confidence in the 
test results while holding computer processing time requirements within 
acceptable limits. Two simple statistics were then computed for each 
model's performance on each specified underlying reliability progress 
path . 

First, the average values (arithmetic means) of the models' estimates 
of the parameter utilized in the character i zat ion of the underlying re- 
liability progress path were computed to provide a measure of the models' 
accuracy. Average values of reliability or failure rate estimates were 
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computed for both the current reliability status at the end of a test 
phase and the forecast reliability status at the end of the following 
test phase. While a summary of these parameter mean value results could 
be presented in tabular form, such mode of presentation is difficult and 
time consuming to comprehend. Therefore, a graphical presentation was 
selected over the tabular form. The graphical presentation permits 
easy qualitative evaluation of a model’s accuracy and recognition of 
estimate trends such as consistent optimistic/pessimistic estimation of 
the reliability progress path. 

For each particular reliability growth model, test structure, and 
specified underlying reliability progress path, a graph was prepared 
depicting the true underlying path, the model’s mean estimated value for 
the reliability or failure rate at the end of each test phase, and the 
model’s mean forecast reliability or failure rate for the end of the 
following test phase. Since a total of I98 different combinations were 
simulated, graphs for all the simulation results are not presented. In 
those cases where no significant change in accuracy or trend results 
occurs graphs are omitted. 

Project managers are concerned with only one ’’replication” of the 
reliability progress path estimation task as opposed to ten, one-hundred, 
or a thousand progress path estimates. Given that the mean performance 
of a reliability growth model is satisfactory, managers need to know if 
the model can be trusted to deliver satisfactory performance on that 
single ’’replication”. This concern equates to the question of varia- 
bility (precision) in the model estimates; i.e., does the model deliver 
’’tight groups” around the mean values of its estimates? To measure 
variability the standard deviation of each model’s reliability or failure 



rate estimates from the mean value performance was computed for all 
phases of all simulations. 

Since standard deviation is a relative measure, it is difficult to 
recognize trend in the performance of the reliability growth models from 
a table of standard deviation statistics. So, variability performance 
of the reliability growth models is presented in tabular form containing 
entries of standard deviation of the model estimates expressed as per- 
centages of the magnitude of the reliability or failure rate values to 
which they correspond. (This statistic is the coefficient of variation 
expressed as a percentage; i.e., the percentage standard error.) Because 
gross performance characteristics of the reliability growth models were 
being evaluated, all percentages were truncated to the whole percentage 
point to facilitate examination of the variability performance tables 
for trends. 

In examining the variability performance of the reliability growth 
models some guideline is desirable. In the field of econometric models 
a percentage standard error of 10 - 15 ^ is the goal given by reference 6. 

In cost estimation modelling a percentage standard error of 15~20% is 
considered satisfactory according to reference 8. A reliability growth 
model that exhibits good mean estimation performance but displays very 
poor variability performance may be useless for practical purposes. On 
the other hand a reliability growth model that produces excellent vari- 
ability performance coupled with poor but consistently biased mean esti- 
mation performance may prove to be very useful. Ultimately, the project 
manager will have to decide if the performance characteristics displayed 
by the reliability growth models examined in the next chapter qualify 
these models for application to his project. 



15 



III. RELIABILITY GROWTH MODEL DESCRIPTIONS 
AND PERFORMANCE RESULTS 



A. CONTINUOUS CUMULATIVE FAILURE RATE RELIABILITY GROWTH MODEL 
1 . Model Description 

The continuous cumulative failure rate model evaluated is of 

the form 



^TT 



( 3 . 1 ) 



where 

= cumulative failure rate of the item thru time TT, 

TT = total accumulated test time for all testing, and 

a, 3 = constants that determine the shape of the model 
estimated reliability progress path. 

Cumulative failure rate may also be thought of as average failure rate 
or character i s t i c failure rate. Under the assumption that a is posi- 
tive, the model relationship is that as test time is accumulated on an 
item (and by implication discrepancies discovered in the item are cor- 
rected) the denominator of the right-hand side of equation 3-1 increases 
and hence, the cumulative failure rate of the item decreases. So reli- 
ability growth is modelled as decreasing failure rate by this model. 
Total accumulated test time TT is measured in whatever units are appro 
priate to the item being tested; i.e., minutes, hours, days, years, etc. 
or mission units. The model is considered continuous since the single 
independent variable TT , total accumulated test time, is continuous 
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as opposed to a discrete variable such as the number of failures experi- 
enced or the number of item modifications accomplished. 



if the failure rate being experienced during any particular 



phase of testing is thought of as the instantaneous failure rate, then 
within the context of reliability growth instantaneous failure rate is 
expected to decrease as testing proceeds from phase to phase just as 
cumulative failure rate is expected to decrease. Since cumulative fail- 
ure rate is a character i zat i on of the failure rate of the item based on 
all the testing that has been accomplished; i.e., an "average" measure 
of failure rate, then cumulative failure rate is expected to be greater 
than the instantaneous failure rate at any point after the initial test 
phase when the two measures are equal. Conversely, if the reliability 
of the item decreases as the acquisition cycle develops (increasing in- 
stantaneous and cumulative failure rate), then cumulative failure rate 
is expected to be less than the instantaneous failure rate being experi- 
enced at any phase after the initial phase of testing. 

A mathematical analysis of the cumulative failure rate model is 
given in reference 2. 

2 • Reliability Testing Procedure 

For the continuous failure rate reliability growth models the 
appropriate reliability testing procedure of a system acquisition model 
is one that is conducted in distinct test phases. These test phases are 
indexed by i with a total of K phases comprising the duration of the pro- 
cedure; i.e., i = 1, 2, 3, , K. [^i th i n each phase of a testing a 

specified number of tests are conducted where the number of tests is 



phase to phase. During each phase of testing an underlying, inherent 



denoted by 




number of tests per phase NT. can vary from 
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failure rate that is unknown to the project managers/contractors is pres- 
ent in the item under test. This underlying failure rate is the instan- 
taneous failure rate of the item for each phase and is denoted as X. . 

I 

Also, during each phase of testing a phase total test time is accumulated 
and labeled as T. . Hence, at the completion of test phase i the total 
accumulated test time TT. on the item is given by 



TT. 

I 




T, 

J 



(3.2) 



for i = 1, 2, 3, K and TTj = . 

For each phase of the testing program a planned test time is es- 
tablished beforehand after which items under test that are still func- 
tioning are judged successful; i.e., non- fa i 1 ures . The planned test 
time for test phase i is denoted as • Therefore, the maximum test 

time that can be accumulated during any given test phase i is given by 
the product NT. x PTT. and occurs only when none of the NT. components 
tested fail. During each phase of the testing the number of failures 
that does occur is recorded as Fj . ^t the end of test phase i the total 
number of failures experienced for the entire testing procedure thru 
phase i is given by 

TF. = 

I 

for i = 1 , 2, 3, . . . , K and TFj = Fj . 

Figure 3*1 is a schematic diagram of the continuous models reli- 
ability testing procedure. The total number of test phases K, the number 
of tests conducted during each test phase NT., and the planned test time 
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CONTINUOUS FAILURE RATE RELIABILITY GROWTH MODELS RELIABILITY TESTING PROCEDURE DIAGRAM 




PTT. at which an item under test is considered successful are determined 

I 

prior to testing. The test time T. accumulated during each test phase , ^ 

the total accumulated tes t~~t i me TTT ^t the end of each test phase, the 

/ 

number of failures Fj experienced during each test phase, and the total 
number of failures TF. accumulate for the entire testing procedure thru 
the end of each test phase are all data collected from the reliability 
testing procedure. The most salient feature of the computer simulation 
of this reliability testing procedure is that the analyst also specifies 
the underlying instantaneous failure rate X. that is effective for each 
phase. Thus an accurate assessment of the accuracy of the model can be 
made because the X. values are known. 

3 • Reliability Testing Procedure Computer Simulation 
a. Summary 

The reliability test procedure appropriate to the continuous 
failure rate reliability growth models was simulated on the Naval Post- 
graduate School W. R. Church Computer Center's IBM 360/67 System utiliz- 
ing standard computer simulation techniques. Reliability testing proce- 
dure design parameters K, NT., and PTT. were specified along with 
various underlying instantaneous failure rate sets. Tests of components 
were then simulated with component success or failure being determined 
for each test based on the appropriate underlying instantaneous failure 
rate X. and the planned test time PTT. . T. , TT. , F., and TF. test 
data were collected as each testing procedure simulation was accomplished. 
Simulations of reliability testing procedures for each specified instan- 
taneous failure rate set were replicated one-hundred times in order to 
assess the continuous failure rate reliability growth models' variability 
performance . 
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After the reliability testing procedure simulations were com- 
pleted, continuous failure rate reliability growth model estimates of the 
underlying cumulative and instantaneous failure rate of the component 
being tested were produced based on the simulation test data by computa- 
tions utilizing ordinary least squares regression techniques. Simple 
mean and variability statistics of these model estimates were then computed 
to permit performance evaluation of the continuous failure rate models, 
b. Detail Description 

The computer simulation of the continuous cumulative and the 
continuous instantaneous failure rate reliability growth models was ini- 
tiated by reading in the following reliability testing procedure design 
specifications; K, the total number of test phases; NT., the number of 
tests to be conducted during each test phase; X., the underlying instan- 
taneous failure rate effective during each test phase; and NSIMS, the 
number of times the specified reliability testing procedure was to be 
simulated. For this thesis NSIMS was always specified as one-hundred 
s imu 1 at i ons . 

Rather than specify the test phase planned test times 
at which components still operating were considered not to have failed 
for each phase, planned test times were calculated for each test phase 
such that 



exp (-X. X PTT.) = 0.7 or 

PTT. = - ^ In 0.7 . (3.4) 

i 

The lifetimes of components tested during each test phase i were taken 
to be exponentially distributed with mean time to failure (MTTF) 1/X.. 
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The planned test times given by equation 3-^ result in each component 
tested having a 10 % chance of surviving the test phase. This figure for 
survival probability of 0.7 was chosen arbitrarily to Insure that some 
component failures were generated. 

The procedure for specifying planned test times admit- 

tedly deviates from actual test procedure design since the calculation 
in equation 3-^ is dependent upon the underlying instantaneous failure 
rate which is unknown in real life testing. However, if a reliability 
testing procedure is being conducted under the assumption of reliability 
growth, planned test times i would naturally be specified as increas- 

ing from test phase to test phase as the expected fa i 1 u re rate decreases. 
If the true underlying failure rate should stagnate or worsen from one 
test phase to the next, then the probability that components fail during 
the next test phase increases since planned test time will have been 
lengthened. Consequently, more failures would generally result than if 
the underlying reliability progress path had followed the expected "nice'* 
growth pattern; i.e., decreasing failure rate. 

After specification data were read into the computer, uniform 

(0,l) random variates were drawn; one corresponding to each component 

tested in each of the K total test phases; i.e., uniform random variates 

were drawn and indexed as U. . for i = 1 , 2 , 3 , . . . , K and j = 1, 2, 3, 

* >J 

... , NT.. For this evaluation the life length of components was assumed 
to be exponentially distributed. The assumption is a strong one, and the 
performance character i s t i cs of the models under other than the exponen- 
tial lifetime distribution assumption might prove to be a fruitful area 
for evaluation. Next, the uniform random variates were transformed into 
exponential lifetimes for the components under test via the probability 



22 



integral transform as given in reference k; i .e. , each component was as- 



signed a test lifetime E. ^ according to 



E. . = - ^ In U. . 
I .J A. I ,j 



(3.5) 



for i = 1, 2, 3> ••• > K and j = 1, 2, 3, ... , NT.. 

The number of failures F. that occurred during each test 
phase was then tallied by comparing the test lifetime j each com- 

ponent with the appropriate planned test time PT’T. such that 



NT. 



F. = /^ 5. where 5.= 

\ ^ J J 



i = l 



1 if E. . < PTT. (Failure) 
• > J • 



0 if E. . > PTT. (Success) 
I ,J = I 



(3.6) 



for i = 1, 2, 3, ... , K. Similarly, the test time T. accumulated for 
each test phase was recorded as 



NT. 



•. = V £. . 

I ^ I ,j 



where e. . = 
i.J 



J=i 



E. . if E. . < PTT. 
I ,J I ,J I 



PTT. if E. . > PTT. 

I I ,J = I 



(3.7) 



for i = 1, 2, 3, ... , K. Finally, the corresponding total failures TF. 
and total test times TT. accumulated thru the end of each test phase 
were tal 1 i ed as 



TF, 



= i: V 



and 



(3.8) 



j = l 



TT 






(3.9) 



for i = 1, 2, 3, ... > K respectively. 
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In keeping with accepted reliability growth modelling prac- 
tice for reliability testing procedures utilizing planned test times, 
in those cases where no failures were experienced during a given test 
phase (F. = O) the number of failures for that test was assigned as one- 
half a failure; i.e., ~ ^ • Similarly, whenever the total failures 

accumulated thru a given test phase were zero (TF. = O) based on the 

i 

original failures per test phase statistics (TF. = Z F. = O), then 

' j = l ■' 

total failures accumulated thru that phase were assigned as one-half a 
failure; i.e., TF. =y . The primary thrust of these modifications is 
that zero failure statistics result in failure rate estimates of 0.0 
which are extremely unrealistic and would virtually never occur under 
the exponential lifetime distribution assumption if infinite planned test 
times were possible. Hence, the one-half failure assignment procedure 
results in optimistic but positive failure rate estimates for those in- 
stances in which no failures are experienced. 

These data constituted the total data gathered from a single 
computer simulation of the continuous model reliability testing proce- 
dure for a single specified underlying reliability progress path; i.e., 
a specified set of failure rates which shall henceforth be referred to 
as a lambda set. This computer simulation was repeated one-hundred times 
(NSIMS = loo) for each specified lambda set where the simulation repli- 
cation is indexed as r = 1, 2, 3, ••• , NSIMS = 100. Therefore, all the 
simulation data are indexed by the simulation number r which has been 
left implicit for simplification; i.e., the simulation results are F. 

TF. , T. , and TT. . Since twenty-eight different lambda sets were 
I , r I j r I > 

utilized for evaluating the continuous failure rate reliability growth 
models with three different number of tests per test phase NT. specified 
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for each lambda set, 8400 reliability testing procedures were simulated 
for the continuous reliability growth models evaluation, 
k . Computer Simulation Data Manipulation 

The continuous cumulative failure rate reliability growth model 
is proposed as a model of the unknown, underlying cumulative failure 
rate Xjj of a component as it proceeds thru a system acquisition cycle 
In each reliability testing procedure computer simulation the true cumu- 
lative failure rate that was in effect at the end of a test phase 

was determined by the test time weighted average of the underlying true 
instantaneous failure rates X. that were in effect thru the end of the 

I 

given test phase. Thus, the true cumulative failure rate was determined 
as 




for i = 1, 2, ••• , It is against this true cumulative failure 

rate that the model's performance was measured. 

At the end of a phase of testing an estimator of the instanta- 
neous failure rate for that phase of testing is given by 




Number of Failures Experienced During Phase i 
Test Time Accumulated During Phase i 



F. 

i_ 

T. 



I 



(3.11) 



for i = 1, 2, 3, ... , K. In turn an estimator of the cumulative fail- 
ure rate thru the end of a phase of testing is given by the test time 
weighted average of the instantaneous failure rate estimates; i.e., 



TT 



= i: ^ 



T. 



r 



j 



( 3 . 12 ) 



J = i 



for i = 1, 2, 3, ... , K. Note that since TT^ = , then = A^ . 
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The estimator of the cumulative failure rate thru the end of any test 
phase may also be stated as 






TF. 

I 

TT. 



(3.13) 



for i = 1, 2, 3, ... > K by the following line of reason; F. = X, T, 

J J J 

by equation 3.11; and therefore, 

i i 

TF. = y ^ F. = y ^ T. by equation 3-8. 

j=l j=l 

So, 

^ TFi 

^TT. " TTT 

I I 



E 

j = l 



X. T. 
J J 



TT. 



E 

j = i 



T. 

TT. 



(3.14) 



for i = 1 , 2 , 3 , ... , K wh i ch is equivalent to equat i on 3.12. Since 
TF^ = F^ and TT^ = T^ , then 





= X, 



TT 




(3.15) 



The estimates of the instantaneous failure rate X. and the 

i 

cumulative failure rate based on data from a reliability 

i 

testing procedure utilizing planned test times PTT. . Therefore, the 
estimates obtained from equations 3.11 and 3.13 are biased estimates, 
the bias being in the pessimistic direction. In particular all of the 
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instantaneous failure rate estimates and the cumulative failure rate 
estimates for the early phases of the testing procedure are biased too 
large in magnitude. To compensate for this estimator bias a bias correc 
t i on factor derived in reference 7 was applied to the instantaneous fail 
ure rate estimates and the cumulative failure rate estimates X, 

I 

such that 



TT. 



F. 



y 

2NT. 



X. 



T. VI + 2NT. 

I ' I 



and 



(3.16) 



TT, 



7 




' / 
2 ^ NT. 

j = l 

i 

•1 + 2 ^ NT.' 

j = 1 



(3.17) 



for i = I, 2, 3, ... , K. 

Next, estimates of a and 6 in the cumulative failure rate 
model, equation 3-1, were obtained using ordinary least squares regres- 
sion techniques. Because ordinary least squares regression requires at 
least two observation points, estimates for a and S could only be 
derived from the model starting at the conclusion of phase two; i.e., 
estimates were derived for phases 1 = 2, 3, ••• > 

If the cumulative failure rate model in equation 3*1 is written 
as 



-a. 



Xtt. = 6, TT. 



( 3 . 18 ) 

and the logarithmic t rans format i on is applied to this equation, then 



1 n X-.- =ln6. - a. InTT. . 
TT. I I I 

I 



(3.19) 
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and 



then equation 3-19 is of the form 



Y. = a. + b. X. . 

I I II 



( 3 . 20 ) 



\ 



Note that the estimated cumulative failure rates \ from equation 

i 

3.17 are utilized in performing the regressions. Applying ordinary 
least squares regression estimates as given in reference 6 the and 

b. estimates are 



T' (X. - X)Y. 
^ J J 



...V. j = ’ 



E <’'j ■ « 

j=l 



0x2 



and 



(3.21) 



•a^ = Y -'^b^X 

I I 



(3.22) 



for i =2, 3, ... J K. The and estimates are then obtained 

by the inverse transformations and = exp ( ). Finally, 
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these a. and S. estimates are utilized in the cumulative failure 

I I 

rate model, equation 3-18, to produce the model estimates of the unknown, 
underlying cumulative failure rate. These estimates are given by 




(3.23) 



for i = 2, 3, ••• , K. For the first test phase the cumulative fail- 

ure rate estimate is simply taken as the estimate given by equation 3 - 17 ; 
i .e. , 




Again, since the reliability testing procedure was simulated one- 



hundred times (NSIMS = lOO) for each lambda set, one-hundred estimates 
of the cumulative failure rate were obtained for each test phase. The 
mean value and standard deviation of these estimates were computed as 



X" 



1 



TT. 



NSIMS 



NSIMS 

E 

r=l 



TT. 



and 



S.D. 



TT. 




NSIMS 



E 





(3.25) 



( 3 . 26 ) 



for i = 1, 2, 3. ... , K. Finally, the measure of variability for the 
cumulative failure rate model was computed as 



P.S.E. 



TT, 




- X 100 




(3.27) 
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for i = 1 , 2 , 3 , • • • , K. 

Because the phase total test time T. and the total accumulated 
test time TT. are random variables which changed from simulation to 
simulation, their mean values were computed as 

NSIMS 

^i " NSIMS S ^i,r (3.28) 

r=l 

NSIMS 

^i = Nsk E ""i.r (3.29) 

r=l 



respectively for i = 1, 2, 3, ..., K. Similarly, the mean value of the 
true underlying test time weighted average cumulative failure rate was 
computed as 



A 



TT. 



I 



NSIMS 

NSIMS E ^TT . 
r=l 



(3.30) 



for i = 1, 2, 3, ... , K. Figure 3.2 is a graphical representation of 
the results obtained from the reliability testing procedure for the con- 
tinuous cumulative failure rate reliability growth model. 



Equations 3.25 ( ) and 3-7 (P.S.E.^ 



) are measures of the 



TT. 



1 

cumulative failure rate reliability growth model's performance in detei — 
mining what a component's reliability status is at the end of a test 
phase. To examine the cumulative model's forecasting capability the 
following procedure was followed. The total accumulated test time TT. 
of the cumulative model, equation 3.18 , is a stochastic (random) vari- 
able. Hence, at the end of any test phase i the total accumulated test 
time for subsequent phases of testing (TT._j_p "*"^i+2’ unknown but 
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MEAN WEIGHTED AVERAGE CUMULATIVE FAILURE RATE 
MEAN MODEL DETERMINED CUMULATIVE FAILURE RATE 
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FIGURE 3.2 

GRAPHICAL REPRESENTATION OF RELIABILITY TESTING PROCEDURE RESULTS FOR CONTINUOUS CUMULATIVE 

FAILURE RATE RELIABILITY GROWTH MODEL 



required if the model is to be used for forecasting. For this evalua- 
tion a one test phase in the future forecast of cumulative failure rate 
was made by approximating the next test phase (i+l) total accumulated 
test time from the current test phase as 



TT. 



i + l 



= TT. 



(NT. 



i + l 






(3.31) 



for i =2, 3, . . . , K. Thus at the end of a given test phase i the 

recorded total accumulated test time TT. for that phase was added to 
the maximum phase total test time that could be accumulated during the 
following test phase i+l; l.e., the number of items to be tested 
times the planned test time PTT._j_j for the next test phase. This pro- 
cedure thus approximated the most optimistic outcome; i.e., no failures 
for the following test phase i+l. The approximated total accumulated 
test time TT. , , for the next phase was then utilized in the current 
phase i cumulative model to obtain a next phase forecast cumulative 
failure rate as 




-a. 



= 6 ,( 



^"i+l) 



(3.32) 



for i = 2, 3, , K- Therefore, forecast cumulative failure rates 

were made for test phases i = 3 , ^ , 5 , • • • , K. No forecast could be 
made for test phase 2 because the model parameters a. and were not 

estimated until test phase 2 was completed. No forecast was made past 
test phase K since there would be no underlying cumulative failure rate 
with which to compare. Finally, forecast cumulative failure rate sta- 
tistics similar to the determined failure rate statistics in equations 
3 . 25 , 3-26, and 3*27 were computed for the simulation replications 
(NSIMS = IOC) as 
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1 

NSIMS 



NSIMS 



E 



FX. 



TT. 



I ,f 



(3.33) 



NSIMS 



S.D.- 



1 



FX^-P \ NSIMS - 1 

i r= 



E 



- FX^^ ) , and (3-3^) 

i , r [ 



S.D. 



FX 



TT. 



P.S.E. 



FX 



X 100 



(3.35) 






TT, 



for i =3, 5, ... , K. 



As a starting point for the evaluation of the continuous failure 
rate models the lambda sets utilized by Jayachandran and Moore in refer- 
ence 5 were adopted as specified underlying instantaneous failure rate 
X. sets for five reliability progress paths. These lambda sets are 
listed in table 3*1 where they are numbered as lambda sets 1 thru 5- 
Lambda sets 1 thru 5 represent “nice” reliability progress paths since, 
although they vary in rate and span of failure rate decay, they have 
smooth exponential like decay patterns (curves). A decaying failure rate 
path is equivalent to a reliability progress path that displays growth. 

Note in table 3-1 that for lambda set 3 for example the specified 
underlying instantaneous failure rate starts at 0.700 for test phase 1, 
decreases to 0.3530 during test phase 2, and finally reaches a value of 
0.0500 for the last test phase, test phase 16. 

Because a primary goal of this thesis was to evaluate the robust- 
ness of the selected reliability growth models, the “nice” lambda sets 1 
thru 5 v/ere augmented with nine anomalous lambda sets numbered 6 thru ]k 
which are also listed in table 3*1« Lambda sets 6 and 7 represent 
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CONTINUOUS RELIABILITY GROWTH MODELS 
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approximately linear underlying reliability progress paths. Lambda sets 
8, 9, and 10 portray underlying reliability progress paths in which 
reliability rapidly attains a certain level and stagnates at that level 
for the duration of the reliability testing procedure (acquisition cycle). 
Lambda sets 11 and 12 display initial "nice" reliability growth followed 
by a period of reliability stagnation and a subsequent continued growth 
pattern. Finally, lambda sets 13 and 1^ represent underlying reliability 
progress paths in which reliability growth is interrupted by a period of 
reliability degradation. 

As indicated in table 3-1 lambda sets 1 thru 14 each represents 
reliability testing procedures of systems acquisition cycles consisting 
of sixteen test phases (K = 16). For very expensive and complex weapons 
systems a sixteen test phase procedure may not be feasible. Therefore, 
lambda sets for a more realistic six test phase (K = 6) cycle were also 
utilized in the computer simulation evaluation. These lambda sets for a 
contracted reliability testing procedure are listed in table 3-2 and 
numbered MODI thru M0D14. Lambda sets MODI thru M0D14 represent underly- 
ing reliability progress patterns that are equivalent to the progress 
patterns characterized by lambda sets 1 thru 14 but which develop during 
a six test phase cycle rather than over a sixteen test phase cycle. 

In order to examine the continuous failure rate reliability 
growth models for a representative set of acquisition programs, procedures 
with items tested per test phase NT. of five, ten, and twenty items were 
simulated utilizing all twenty-eight lambda sets; i.e., NT. = 5, 10, and 
20 for i = 1, 2, 3. ••• , 16 or i = 1, 2, 3. 4, 5, 6 as appropriate. 

Hence, 28 x 3 x 100 = 8400 reliability testing procedures were simulated 
for the continuous failure rate reliability growth models evaluation. 
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MODIFIED LAMBDA SETS 
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5 . Model Performance 



a. Accuracy Performance 

Figures 3-3 thru 3-^1 present selected cases of the continu- 
ous failure rate reliability growth model’s capability to determine and 
forecast the unknown, underlying cumulative failure rate progress path 
during an acquisition cycle. These graphs depict the mean test time 

weighted average true underlying cumulative failure rate Xjj progress 

i 

path from equation 3-30 ( , solid line), the mean model determined 

cumulative failure rate Xjj for each test phase from equation 3-25 (0, 

circles), and the mean model forecast cumulative failure rate "f^or 

i 

the third thru the final test phase from equation 3*33 (X , crosses) 
plotted versus the mean total accumulated test time TT. from equation 
3 . 29 . Note that the point plotted for the test phase 1 mean model deter- 
mined cumulative failure rate is not a mean model determined estimate; 
but rather, it is the mean estimate of the cumulative failure rate (and 
i ns tantaneous failure rate since the estimate is for test phase l) given 
by equation 3*24 ( Xjj = X^). So the accuracy of the reliability esti- 



mator of equation 3*2A may also be observed to a degree by examining the 
results for test phase 1. 

To illustrate these quantities graphed in the accuracy perfor 
mance figures note on figure 3-11 that at approximately 82.00 total accu- 



mulated test time units the mean weighted cumulative failure rate Xjj 



16 



was 0.30 while the mean model determined cumulative failure rate Xjj 



16 



was approximately 0.45 and the mean model forecast cumulative failure 
rate 

points on the plot, they represent the simulation results for the six- 
teenth phase of a reliability testing procedure wherein five tests per 



-- was approximately 0.50. Since these values were the last 

“16 
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phase were conducted and for which lambda set 7 from table 3-1 specified 
the unknown, underlying reliability progress path. Also, note that the 
first mean model forecast cumulative failure rate point corresponds to 
the third mean model determined cumulative failure rate plotted; i .e. , 
the first model forecast cumulative failure rate was for test phase 3 
(i =3). Finally, for test phase 1 the point plotted for the mean model 
determined cumulative failure rate is approximately 0.73 which is actual- 
ly the mean value of the estimate produced by the instantaneous failure 
rate estimator of equation 3-2^. Notice that the mean test time weighted 
average cumulative failure rate at test phase 1 is 0.70 which corresponds 
to the specified underlying instantaneous failure rate for this phase 
from table 3*1- This equivalence is explained by equation 3- 15- 

Figures 3»3 thru 3*9 and 3*25 thru 3-29 indicate that for 
"nice" underlying reliability progress paths the cumulative failure rate 
model displays very accurate performance with slight pessimistic bias 
(higher failure rate estimates). "Least data" graphs; i.e., five tests 
per phase cases (NT. = 5) are generally displayed. Ten and twenty tests 
per phase (NT. = 10/20) performance was as good as or better than five 
tests per phase for all cases in which only the NT. = 5 graphs are 
shown. This consistency of performance is displayed for lambda set 3, 

NT. = 5/10/20 sequence in figures 3*5, 3-6, and 3*7> Forecasting perfor- 
mance displays a general trend of overly pessimistic forecasts for the 
first two or three phases improving rapidly after four or five forecasts 
have been made. 

Figures 3-10 thru 3-13, 3-30, and 3-31 reveal that the cumu- 
lative failure rate model has some difficulty in tracking linear under- 
lying reliability growth patterns even when a relatively large amount 



38 



of data are available (figure 3- 13, NT. = 20) . In all the linear cases 
both the model determined and model forecast mean failure rates diverge 
from the actual cumulative failure rate. Performance does not improve 
as the number of components tested per phase is increased as evidenced 
by the lambda set 7, NT. = 5/10/20 sequence performance in figures 3.11, 
3 . 12 , and 3.13- 

Figures 3-1^ thru 3-20 and 3-32 thru 3-36 indicate that the 
cumulative failure rate model copes with permanently stagnated underly- 
ing reliability progress paths easily. The figure series 3*1^ thru 3-16, 
3.18 thru 3 - 20 , and 3-32 thru 3 - 3 ^ contrast the accuracy performance for 
increasing number of test per phase (NT. = 5/10/20). Again, forecasting 
performance is initially very poor but quickly improves after three of 
four forecasts have been made. 

In situations of the underlying reliability progress path 
experiencing temporary stagnation figures 3-21, 3-22, and 3-37 thru 3-39 
reveal that the cumulative failure rate model, while picking up the 
stagnation credibly, has difficulty in accurately modelling the cumula- 
tive reliability improvement following the period of temporary stagna- 
tion. During this post stagnation improvement phase, the cumulative 
model's determined and forecast failure rate estimates diverge on the 
pessimistic side from the true underlying cumulative failure rate path 
much as they did in the linear underlying progress path cases. Figures 
3.38 and 3.39 show that this trait is not relieved by gathering more 
test data per test phase of the reliability testing procedure (NT. = 5 
vs. NT. = 20) . 
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FIGURE 3.3 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 1; 16 PHASES, 5 TESTS/PHASE 
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FIGURE 3.4 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 2: 16 PHASES, 5 TESTS/PHASE 
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FIGURE 3.5 

CUMULATIVE RELiRBILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET 3: 16 PHASES, 5 TESTS/PHASE 
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FIGURE 3.6 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 3: 16 PHASES, 10 TESTS/PHASE 
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FIGURE 3.7 

CUMULATIVE RELIflBILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET 3; 16 PHASES. 20 TESTS/PHASE 
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FIGURE 3.8 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET M; 16 PHASES. 5 TESTS/PHASE 
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FIGURE 3.9 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 5; 16 PHASES, 5 TESTS/PHASE 
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FIGURE 3. 10 

CUMULPTIVE RELIRBILITT GROWTH MODEL PERFORMPNCE 
LPMBOP SET 6; 16 PHASES, 5 TESTS/PHflSE 
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FIGURE 3.11 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 7; 16 PHASES. 5 TEST5/FHASE 
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FIGURE 3. 12 

CUMULATIVE RELIflBILITT GROWTH MODEL PERFORMANCE 
LflMBOfl SET 7; 16 PHASES, 10 TESTS/PHASE 
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FIGURE 3. 13 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 7: 16 PHASES, 20 TESTS/PHASE 



50 



■ '-C* 




MEAN CUMULATIVE FAILURE RATE 

.00 0.10 0.20 0.30 O.HO 0.50 0.60 0.70 0.60 0.90 l.OO 



*^.00 



X 




X X 



TJ U 



^ ^ » 



h e 



e » » 



. I MEAN WEIGHTED CUMULATIVE FAILURE RATE 
O I MEAN MODEL DETERMINED CUMULATIVE FAILURE RATE 
X t MEAN MODEL FORECAST CUMULATIVE FAILURE RATE 



s'. 00 I'o.oo ik.oo ioToo iiToo 30.00 ik.oo 
TOTAL ACCUMULATED TEST TIME 

FIGURE 3.14 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 8: 16 PHASES, 5 TESTS/PHASE 
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FIGURE 3. 15 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 8: 16 PHASES. 10 TESTS/PHASE 
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FIGURE 3.16 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 8; 16 PHASES, 20 TESTS/PHASE 
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FIGURE 3.17 

CUMULATIVE RELIABILITT GROWTH MOOEL PERFORMANCE 
LAMBDA SET 9: 16 PHASES, 5 TESTS/PHASE 
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FIGURE 3. 18 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 10; 16 PHASES, 5 TESTS/PHA3E 
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FIGURE 3.19 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 10; 16 PHASES, 10 TESTS/PHASE 
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FIGURE 3.20 

CUMULATIVE RELIfiBILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET 10: 16 PHASES, 20 TESTS/PHASE 
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FIGURE 3.21 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 11; 16 PHASES. 5 TESTS/PHASE 
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FIGURE 3.22 

CUMULATIVE RELIABILITY GRO'/lTH MODEL PERFORMANCE 
LAMBDA SET 12: 16 PHASES, 5 TESTS/PHASE 
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FIGURE 3.23 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 13: 16 PHASES, 5 TESTS/PHASE 
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FIGURE 3.24 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 14; 16 PHASES, 5 TESTS/PHRSE 
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FIGURE 3.25 

CUMULATIVE RELIABILITY GROWTH MOOEL PERFORMANCE 
LAMBDA SET MODI: 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.26 

CUMULATIVE RELIfiBILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET MOD2; 6 PHASES. 5 TESTS/PHASE 
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FIGURE 3.27 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MOOS: 6 PHASES. 5 TESTS/PHASE 
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FIGURE 3.28 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MODU: 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.29 

CUMULATIVE RELIABILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET MODS: 6 PHASES, 5 TESTS/PHASE 



66 



» 



i 



o 

o 



« t MEAN WEIGHTED CUMULATIVE FhILURE RATE 
O I MEAN model determined CUMULATIVE FAILURE RATE 
X * MEAN MODEL FORECAST CUMULATIVE FAILURE RATE 



o 

oo 

o 



X 




o 

o 

^^dToo 3'. 00 s'. 00 9'. 00 12.00 lb. 00 la.oo 21 . 00 

TOTAL ACCUMULATED TEST TIME 



FIGURE 3.30 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MODS: 6 PHASES, 5 TESTS/PHRSE 
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FIGURE 3. 31 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MOD?: 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.32 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MOOS: 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.33 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MODS; S PHASES, 10 TESTS/PHASE 
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FIGURE 3.34 

CUMULfiTIVE RELIFIBILITT GROWTH MODEL PERFORMANCE 
LRMBDR SET MODS: 6 PHASES, 20 TESTS/PHASE 
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FIGURE 3.35 

cumulative relirbilitt growth model performance 

LAMBDA SET M0D9; 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.36 

CUMULATIVE RELIABILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET MODIO: 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.37 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MODll; 6 PHASES. 5 TESTS/PHASE 
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FIGURE 3:38 

CUMULATIVE RELIABILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET M0D12: 6 PHASES. 5 TESTS/PHASE 
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FIGURE 3.39 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET M0DI2; 6 PHASES. 20 TESTS/PHASE 
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FIGURE 3.40 

CUMULATIVE RELIflBILITT GROWTH MODEL PERFORMANCE 
lambda set M0D13; 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.41 

CUMULATIVE RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET M0D14: 6 PHASES, 5 TESTS/PHASE 
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Figures 3-23, 3-2h, 3*^0, and 3-^1 indicate that in instances 
where reliability growth progress patterns are interrupted by a period of 
reliability degradation the cumulative failure rate model lags behind the 
true underlying cumulative failure rate path with both its determined and 
forecast failure rate estimates. While this characteristic might be dis- 
missed as the result of the "smoothing" nature of the model, the model's 
failure rate estimates are plotted with the mean test time weighted aver- 
age cumulative failure rate which is itself a "smoothed" quantity. Hence, 
the model cannot be excused from failing to accurately portray the tem- 
porary reliability progress decay case. "Least data" cases (NT. = 5) are 
presented and the performance was no worse for the ten and twenty test 
per phase simulations. The cumulative model failure rate estimates did 
display reliability progress patterns that resembled dampened versions of 
the true underlying cumulative failure rate paths, and the model's esti- 
mates were reasonably accurate for the final cumulative failure rate 
status . 



b. Variability (Precision) Performance 

Tables 3-3 thru 3-8 present the continuous cumulative failure 
rate reliability growth model's variability performance for determining 
the cumulative failure rate status of a component in a systems acquisi- 
tion cycle. Entries in these tables (P.S.E.-^ ) were determined by 

^TT. 



equation 3-27. Tables 3-9 thru 3-1^ present the 
model's variability performance for forecasting 
rate status of a component one test phase ahead 
phase (i). Entries in these tables (P.S.E."^ 



cumulative failure rate 
the cumulative failure 
(i+1) of the current test 
) were determined by 



equation 3-35. 



79 



From figure 3-11 the mean model determined cumulative failure 

rate for the sixteenth phase of testing for lambda set 7 using five tests 

per phase was read as X" = 0.45. In table 3*3 under lambda set 7, 

“l6 

phase 16 a value of 25% is entered for P.S.E.'^ . Therefore, the 

TTig 

standard deviation of the mean model determined cumulative failure 

rate X* = 0.45 over one-hundred simulations (NSIMS = lOO) was 
"l6 

P . S . E 

- ?T„=<-ior— ^ 

TTi^ 16 

from equation 1 . 11 . 

As noted in Chapter II econometricians and cost estimators 
set goals of 10% to 20% for the percentage standard error measure of vari- 
ability performance of their models. Since gross performance is being 
examined in this thesis, less stringent bench marks may be appropriate 
for evaluating the variability performance of the reliability growth 
models. 

If a goal of 30 % is set for the percentage standard error 
measure of variability performance, then a quick method of examining a 
model's variability performance is to determine in each variability per- 
formance table the first test phase at which the percentage standard 
error for all lambda sets (failure rate sets) meets the 30 % goal. The 
earlier the test phase number at which this criterion is achieved the 
more confidence can be placed in the precision of the model. As with 
any sampling statistic, increasing sample size is expected to reduce 
variability. In the reliability testing procedure as the number of tests 
per phase HT. is increased from five to ten to twenty the test phase 
number at which the 30% percentage standard error goal is achieved is 
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therefore expected to occur earlier in testing. 

Tables 3-3 thru 3-1^ list the variability performance of the 
continuous cumulative failure rate reliability growth model. In tables 
3 . 3 , 3 .^, and 3«5 the cumulative failure rate model's variability perfor- 
mance for determined cumulative failure rate and s ixteen phase lambda sets 
achieves the 30 % goal at phase 10 for NT. = 5, at phase 6 for NT. = 10, 
and at phase 2 for NT. = 20. The percentage standard error listed for 
phase 1 in any of the tables is for the failure rate estimates given by 
equation 3*24 vice for the cumulative model estimates which start with 
test phase 2. As can be seen the equation 3-24 estimates are subject to 
a great deal of variation. Tables 3*6, 3*7, and 3*8 show that in the 
case of the contracted six test phase cycles the cumulative model's vari- 
ability performance for determined cumulative failure rate fails to 
achieve the 30 % goal for NT. = 5 and achieves the 30 % goal at phase 5 
for NT. = 10 and phase 3 for NT. = 20. 

i I 

In tables 3-9, 3-10, and 3-11 the cumulative failure rate 
model's variability performance for forecast cumulative failure rate in 
sixteen test phase cycles achieves the 30^ percentage standard error 
goal at phase 12 for NT. = 5, at phase 7 for NT. = 10, and at phase 5 for 
NT. = 20. Of course forecasts are made for phases 3 and on only. Tables 
3 . 12 , 3«13, and 3-14 show that for the six phase lambda sets the cumula- 
tive model fails to achieve the 30^ percentage standard error goal for 
variability performance in forecasting cumulative failure rates. The 
difficulties arise for lambda sets M007, M0D12, and M0014. These are the 
lambda sets for which the mean test time weighted average underlying 
cumulative failure rate decreased most rapidly to approximately 0.20 for 
phase 6 in all cases, and these lambda sets are the ones for which the 
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cumulative model displayed the most difficulty in tracking the underly- 
ing cumulative failure rate progress path. Refer to figures 3-31, 3*39, 
and 3-^1 • Note in table 3-1^ (NT. = 20) that the 30% goal was achieved 
in phase 5 but lost in phase 6 for the three lambda sets enumerated 
above. If lambda sets M0D7, M0D12, and M0D14 were not considered, then 
the 30% percentage standard error goal was achieved in phase 6 for NT. = 
10 . 

B. CONTINUOUS INSTANTANEOUS FAILURE RATE RELIABILITY GROWTH MODEL 
1 . Model Description 

The continuous instantaneous failure rate reliability growth 
model evaluated is of the form 

TT^ 



where 

X = instantaneous failure rate, 

TT = total accumulated test time for all testing, and 

a,b = model parameters that determine the shape of the model 
estimated instantaneous failure rate reliability 
progress path. 

Total accumulated test time TT, the independent variable, is the same 
quantity utilized in the continuous cumulative failure rate reliability 
growth model, equation 3.1. Instead of the cumulative failure rate 
X.|..p modelled by equation 3-1, the instantaneous model of equation 3*37 
models the unknown, underlying instantaneous failure rate X. . There- 
fore, rather than producing estimates of the underlying test time 

weighted average cumulative failure rate X.^..^. , the instantaneous model 

i 

was used to estimate the actual test phase instantaneous failure rates 
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X. specified as inputs to the reliability testing procedure computer 
simulation. These are the failure rates of lambda sets contained in 
tab 1 es 3-1 and 3 • 2 . 

As can be seen in equation 3*37 and similar to the cumulative 
failure rate model, as total accumulated test time increases instanta- 
neous failure rate is modelled as decreasing. Thus, reliability of com 
ponents is portrayed as increasing in general. Assumptions regarding 
the signs of the values for the model parameters a and b in the instan- 
taneous model are not as obvious as the assumptions might be for the 
signs of the model parameters a and Q in the cumulative failure rate 
model . 

A mathematical analysis of the instantaneous failure rate model 
is given in reference 2. A Monte Carlo simulation evaluation and com- 
parison of the instantaneous failure rate model is contained in refer- 
ence 5. 

2 . Reliability Testing Procedure 

The reliability testing procedure of a system acquisition cycle 
appropriate to the continuous instantaneous failure rate reliability 
growth model is identical to the procedure described for the cumulative 
failure rate model in chapter I I I .A.2. Therefore, the test data nec.es- 
sary for evaluation of the instantaneous model were collected simul- 
taneously with the collection of data for the evaluation of the cumula- 
tive model in a reliability testing procedure as graphically depicted i 
figure 3<1. No additional test results are required for evaluation of 
the instantaneous failure rate model. 

3 . Reliability Testing Procedure Computer Simulation 

Since the reliability testing procedure applicable to the con- 
tinuous and instantaneous failure rate models are identical, data for 
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evaluating both models were collected from a single computer simulation 
program as described in chapter I II. A. 3. Features of the reliability 
testing procedure computer simulation such as phase planned test time 
PTT. determination and assignment of one-half a failure in those in- 
stances when no failures occurred during a test phase were in affect 
for the evaluation of both the failure rate models. Also, the same 
twenty-eight lambda sets (failure rate sets) given in tables 3*1 and 
3.2 and the same number of simulation replications (NSIMS = lOO) were 
utilized for both model evaluations. 

k . Computer Simulation Data Manipulation 

The continuous instantaneous failure rate reliability growth 
model of equation 3*37 may be written as 

-a . 

A. = (l-a.)b.(TT.) ' . ( 3 . 38 ) 



As noted in equation 3*16 an estimate of the unknown, underlying instan- 
taneous failure rate of a component for any given test phase of a reli- 
ability testing procedure corrected for planned test time bias is given 
by 



2NT. F. 

^i " 1 + 2NT. ’ T7 

I I 



(3.39) 



for i = 1, 2, 3, ••• , K where T. is the total test time accumulated 
for the number of components NT. tested during the phase and F. is 
the number of component failures occurring during the phase. 

If the logarithmic transformat i on is applied to equation 3-38, 

then 

lnX.=ln(l-a.)b. -a.lnTT. . (3.^0) 
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To obtain the ordinary least squares regression estimates for a. and 
b. the data pairs (In X., In TT.) are employed from the results of the 
reliability testing procedure computer simulation. Let 



Y. = In X. , 

1 I 



X. = In TT. , 

I I 



5- = i y; . 

I I i—j j 

j=i 



and 



1 



I 



X. = V X. 

‘ ' Zv j 

j=i 



for i = 1, 2, 3, ... , K. Then equation 3.^0 becomes 



Y. = In b.(l-a.) - a.X, 

I II II 



(3.41) 



for i = 1 , 2 , 3 , . • . , K. By the derivation presented in Appendix A the 
ordinary least squares regression estimates for the instantaneous failure 
rate model parameters a. and b. are 



a . 



V X.Y. - Y. V X. 

j j I j 

= J^] 



A 



i=l 



D - E ’'j' 

j=l j=l 



and 



(3.42) 




1 






exp (Y. +'a'. X.) 



(3.43) 



for i = 2 , 3 , 4, ... , K. Note that as with the cumulative failure rate 
model the regression requires at a minimum observations on the results 
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.1 

i 



i 



of two test phases; thus, model parameter estimates are made for the 
t h 

second thru the K test phase. The instantaneous failure rate estimate 
given by equation 3-39 was used for the first phase of testing; i.e., 




2NT^ 

1 + 2NTj 




( 3 . 4 ^) 



As the instantaneous failure rate model parameter estimates were 
obtained, they were applied to the model in equation 3-38 to obtain model 
determined estimates for the instantaneous failure rate reliability 
status at the end of each phase of testing; i.e., 

' (3.45) 

i 

for i = 2, 3, 4, ... , K. Again, the simulation index r is left impli- 
cit (r = 1 , 2, 3, ... , NSIMS = 100) . 

Upon completion of all computer simulations (NSIMS = lOO) for 
each specified lambda set, performance statistics were computed as 




NSIMS 

NSIMS 23 ^T. 

r=l 



(3.46) 



S 







NSIMS-1 



NSIMS 



T. 



r=l 



I , r 



1 



and 



P.S.E. 




S.D. 



T. 



X 100 




(3.47) 



(3.48) 



I 

for i = 1, 2, 3> ... > K. Also, in order to evaluate the forecasting 
performance of the instantaneous failure rate model the same method of 
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computing an estimate of the total accumulated test time for a one test 
phase in the future failure rate estimate utilized for the cumulative 
failure rate model in equation 3-31 was utilized for the instantaneous 
failure rate model forecasts. Hence, 



-a , 



F\- = (TT.^,) where 



T. 



i + 1 



i i i + 1 



TT.^i = TT. + (NT. , X PTT.^,) 

1 + ] I 1 + 1 1 + 1 



( 3 .^ 3 ) 

(3-50) 



for i = 2, 3, ••• , K. Note again that forecasts were made for test 
phases 3 thru K as was the case with the cumulative failure rate model. 
Forecast statistics were then computed as 



NSIMS 



FX 



T. NSIMS 

I 






(3.51) 



r=l 



I , r 



S.D.' 



4 



NSIMS 



F\ TNSIMS-1 

i r=l 



. and (3.52) 



I , r 



S.D. 



FX 



T. 



P.S.E..^ 

P A-, 



X 100 



(3.53) 






for i =3, k, 5, ... , K. 

5 . Model Performance 

a. Accuracy Performance 

Figures 3.^2 thru 3.77 depict the accuracy performance of 
the continuous instantaneous failure rate reliability growth model for 
selected tests per phase NT. cases of the lambda sets listed in tables 
3.1 and 3.2. In each*graph the specified underlying instantaneous 
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failure rate A.. from table 3*1 or 3-2 ( , solid line), the mean 

model determined instantaneous failure rate from equation 3-^6 

i 

(0 , circles), and the mean model forecast instantaneous failure rate 

FA-j. from equation 3*51 (X , crosses) are all plotted against the mean 
i 

total accumulated test time TT. from equation 3-29 for each phase of 
the specified reliability testing procedure of the acquisition cycle. 
While the specified underlying instantaneous failure rate is plotted as 
a smooth, continuous line for purposes of contrast, it should be noted 
from both figures 3-1 and 3*2 and the testing procedure design that the 
specified underlying instantaneous failure rate is actually a step func- 
tion in shape with discontinuities occurring at the end of each test 
phase. 

Examining figure 3-^7 for example at a mean total accumulated 
test time of approximately 205-0 test time units the specified instanta- 
neous failure rate is 0.30, the mean model determined instantaneous 
failure rate is approximately 0.24, and the mean model forecast instanta- 
neous failure rate is approximately 0.19. As noted on the graph the 
accuracy performance plotted is for the lambda set 6, 16 test phase, 20 
tests per phase reliability testing procedure simulation. Since the 
point examined is for the last test phase (i = K = 16), table 3-1 shows 
that for lambda set 6, test phase 16 the specified instantaneous failure 
rate is 0.30 as expected. 

The instantaneous failure rate model frequently yielded mean 
forecast failure rates that were "of f- the-sca 1 e" of the accuracy perfor- 
mance graphs; i.e., mean forecast failure rates greater than 1.0. These 
off-the-sca 1 e mean estimates ranged from failure rates only slightly 
greater than 1.0 to mean estimated failure rates of magnitudes in excess 
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of 10 . The extreme values occurred only for the first forecast and 
occasionally for the second forecast (test phases 3 and 4). Regardless 
of the magnitude of these of f - the-sca 1 e forecast failure rates, all 
points of magnitude greater than 1.0 were plotted at 1.0. 

Figures 3*42 thru 3-46 and 3-60 thru 3*64 display the instan- 
taneous failure rate model's performance for the "nice" underlying fail- 
ure rate progress paths. The model, while discerning the shape of 
these "nice" underlying failure rate patterns, is consistently optimistic 
in both determining the current failure rate status and forecasting the 
next test phase failure rate, especially in the case of the longer six- 
teen phase reliability testing procedures. The model performs more 
accurately for the more volatile six test phase procedures (figures 3-60 
thru 3*64) than for the protracted sixteen test phase procedures (figures 
3.42 thru 3-46). In fact mean determined failure rate performance is 
excellent for the shorter six test phase cases. Forecasting accuracy 
typically is of f- the-scal e for the first test phase improving rapidly 
by the third or fourth test phase. Once stabilized, the mean forecast 
failure rate generally is biased optimistically as expected from the 
method utilized to produce forecasts (equations 3-49 and 3*50). 

In figures 3-47, 3-48, and 3-65 thru 3-67 accuracy of the 
instantaneous failure rate model is displayed for essentially linear 
underlying failure rate progress paths. Again, mean determined failure 
rate performance shows that the model detects the linear shape of the 
progress path with reasonable accuracy but provides consistently opti- 
mistic estimates of the underlying instantaneous failure rate. Forecast- 
ing accuracy, while consistent in the sixteen test phase procedures, is 
very poor being only around 50?-60^ of the magnitude of the underlying 
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instantaneous failure rate for the majority of the test phases until 
the final one or two phases. For the six phase procedures forecasting 
performance is extremely erratic as can be seen in figures 3-65 thru 
3.67. In fact forecasting performance actually degrades as more test 
information becomes available in the case of lambda set MODS; i.e., as 
the number of tests per phase increases from five to twenty (NT. = 5 
vs. NT. = 20, figure 3.65 vs. 3.66). Also, figure 3-67 is for a twenty 
tests per phase (NT. = 20) procedure; and therefore, in keeping with the 
convention established for presenting accuracy performance graphs this 
figure shows the best performance of the instantaneous failure rate 
model for lambda set M0D7. The five and ten tests per phase performance 
for lambda set M0D7 were worse than the performance shown in figure 3.67* 
Figures 3-^9 thru 3-52 and 3.69 thru 3.17 portray the in- 
stantaneous failure rate model's accuracy performance for permanently 
stagnated reliability status cases. The most vivid contrast for these 
cases is between situations in which failure rate stagnates at a high 
value (X. = O. 7 O; figures 3-^9, 3-68, and 3.69) versus stagnation at a 
low rate (X. = 0.05; figures 3.52 and 3. 71). When the underlying failure 
rate stagnates at a high value, the instantaneous model's mean determined 
failure rate estimates are very erratic (X. = O. 7 O; figures 3.^9 and 
3 . 68 ) and the mean forecast estimates are virtually useless from a magni- 
tude consideration being, even once stabilized, approximately ^ 0 ^- 50 ^ 
of the magnitude of the true underlying instantaneous failure rates. 

Note that figure 3.^9 is for a "best case" situation of NT. = 20. Fore- 
casts at best give an indication of stagnation when the stagnation occurs 
at a high true underlying value. In the case of lambda set M0D8 deter- 
mined estimated failure rate improves significantly when more testing 
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is performed; i.e., figure 3-69 versus figure 3-68 (NT, = 20 vs. NT. = 5) • 
On the other hand, when the true underlying failure rate stagnates at a 
low point, both the model's determined and forecast accuracy performance 
appear to be excellent (A. = 0.05, figures 3-52 and 3-71, NT. = 5). 

Figures 3.50, 3-51, and 3-70 show for an intermediate stagnation level 
(A. = 0 . 30 ) how determined and forecast failure rate performance of the 
instantaneous model transitions from poor to excellent. 

The contrast between the underlying failure rate permanent 
stagnation cases points out a general performance character i st ic of the 
instantaneous failure rate model. Because the model produces consistent 
optimistically biased determined and forecast failure rate estimates, as 
the true underlying failure rate progress path approaches low failure 
rates the model's accuracy performance improves significantly by neces- 
sity since the estimates become "sandwiched" between the low true under- 
lying failure rate and 0.0. This "closing of the brackets" effect may 
also produce favorable variability performance for the instantaneous 
failure rate model. 

Figures 3-53, 3-5^, 3-72, and 3-73 present the accuracy per- 
formance of the instantaneous failure rate model in the case of true 
reliability status progress interrupted by a period of stagnation. Only 
"best case" (NT. = 20) graphs are shown. The model's mean determined 
instantaneous failure rate performance is fair for these cases and con- 
tinues to exhibit consistent optimistic bias. Mean forecast failure 
rates generally are indicative of the underlying failure rate path trend 
only and are significantly off the mark in forecasting instantaneous 
failure rate magnitude. The "kick-down" in the mean forecast of the 
failure rate for the first phase after the period of stagnation in figure 
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3,53 (test phase 11 at approximately 130. 0 total accumulated test time 
units) is more pronounced in the five and ten tests per phase (NT, = 5, 

10) cases for lambda sets 11 and 12. 

For situations of temporary reliability status degradation 
figures 3-55 thru 3-59 snd 3-7^ thru 3-77 reveal that in these "worst 
situation" cases the instantaneous failure model performs surprisingly 
well in discerning the shape of the true underlying failure rate path, 
especially for the twenty tests per phase procedures (NT. = 20; figures 
3 . 57 , 3 . 59 , 3 . 75 , and 3 . 77 )* During the period of failure rats degrada- 
tion, the instantaneous model tends to produce determined failure rate 
estimates that overstate the degree of degradation. This is especially 
true in the "least data" cases (NT. = 5, figures 3-55 and 3-58) of the 
sixteen phase testing procedures. This character i st i c is not as pro- 
nounced for the six test phase procedures (NT. = 5, figures 3-7^ and 
3 . 78 ). The model's mean forecast instantaneous failure rate estimates 
remain optimistic throughout the periods of degradation, but reflect the 
shape of the true underlying failure rate progress path with very credible 
fidelity. The figure series 3‘55 thru 3*57 indicates how mean determined 
failure rate estimating performance improves with additional testing 
(NT. = 5, 10, and 20) while mean forecast failure rate estimating perfor- 
mance is very stable after the third forecast for all test per phase 
sizes. Again, note in figures 3-59 2 nd 3-77 that when the true instanta- 
neous failure rate has decreased to a small magnitude (test phase 16), 
the instantaneous model's determined and forecast failure rate accuracy 
displays significant improvement. 

b. Variability (Precision) Performance 

The variability performance of the continuous instantaneous 
failure rate model is presented in tables 3*15 thru 3-26. Entries in 
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FIGURE 3.42 

INSTRNTflNEOUS RELIRBILITT GROWTH MODEL PEPFORMRNCE 
LRMBOFl SET 1; 16 PHASES, 5 TESTS/PHflSE 
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FIGURE 3. 43 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 2: 16 PHASES. 20 TESTS/^HASE 
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FIGURE 3.44 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 3: 16 PHASES, 5 TESTS/PHASE 
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FIGURE 3.45 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 4: 16 PHASES. 5 TESTS/PHASE 
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FIGURE 3.46 

INSTANTANEOUS RELIABILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET 5: 16 PHASES, 5 TESTS/PHASE 
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FIGURE 3.47 

INSTANTANEOUS RELIABILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET 6: 16 PHASES, 20 TESTS/PHASE 
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FIGURE 3.48 

INSTaNTflNEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 7; 16 PHASES, 10 TESTS/PHASE 
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. FIGURE 3.49 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 8; 16 PHASES, 20 TESTS/PHASE 
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FIGURE 3.50 

INSTRNTRNEOUS RELlflBILITT GROWTH MODEL PERFORMANCE 
LRMBOfl SET 9; 16 PHASES, 5 TESTS/PHflSE 
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FIGURE 3.51 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 9; 16 PHASES, 20 TESTS/PHASE 
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FIGURE 3.52 

iNSTflNTPNEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 10: 16 PHASES, 5 TESTS/PHASE 
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FIGURE 3.53 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 11; 16 PHASES, 20 TESTS/PHASE 
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FIGURE 3.54 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 12: 16 PHASES, 20 TESTS/PHRSE 
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FIGURE 3.55 

INSTANTANEOUS RELIABILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET 13: 16 PHASES, 5 TESTS/PHASE 
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FIGURE 3.56 

INSTANTANEOUS RELIABILITT GROWTH MODEL PERFORMANCE 
LPMBOA SET 13; 16 PHASES, 10 TESTS/PHASE 
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FIGURE 3.57 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 13; 16 PHASES, 20 TESTS/PHASE 
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FIGURE 3.58 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET IM: IS PHASES, 5 TESTS/PHASE 
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FIGURE 3.59 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET 14; 16 PHASES. 20 TESTS/PHASE 
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FIGURE 3.60 

INSTANTANEOUS RELIASILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET MODI; 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3. 61 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET M0D2; 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.62 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MODS: 6 PHASES. 5 TESTS/PHRSE 
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FIGURE 3.63 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MODM; 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.64 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MODS; 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.65 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MODS; 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.66 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET M0D6; 6 PHASES, 20 TESTS/PHRSE 
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FIGURE 3.67 

INSTRNTflNEOUS RELIflBILiTT GROWTH MODEL PERFORMANCE 
LRMBOfl SET MG07: 6 PHASES, 20 TESTS/PHASE 
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FIGURE 3.68 



INSTANTANEOUS RELIABILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET MODS: 6 PHASES. 5 TESTS/PHASE 
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FIGURE 3.69 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MODS; 6 PHASES, 20 TESTS/PHASE 
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FIGURE 3.70 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MODS: 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3. 71 

INSTANTANEOUS RELIABILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET MODIO; 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.72 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MOOll; 6 PHASES, 20 TESTS/PHASE 
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FIGURE 3.73 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET M0D12; 6 PHASES. 20 TESTS/PHASE 
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FIGURE 3.74 

INSTANTANEOUS RELIA0IL1TT GROWTH MODEL PERFORMANCE 
LAMBDA SET MOD13: 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.75 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET M0013; 6 PHASES, 20 TESTS/PHASE 
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FIGURE 3.76 

INSTANTANEOUS RELIABILITY GROWTH MODEL PERFORMANCE 
LAMBDA SET MOOIM; 6 PHASES, 5 TESTS/PHASE 
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FIGURE 3.77 

INSTANTANEOUS RELIABILITT GROWTH MODEL PERFORMANCE 
LAMBDA SET MODIM; 6 PHASES, 20 TESTS/PHASE 
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these tables are the percentage standard errors as computed in equations 
3.48 and 3- 53- From figure 3*47 the mean model determined instantaneous 
failure rate was read as 0.24 for the sixteenth test phase of the lambda 
set 6, 16 test phase, 20 tests per phase reliability testing procedure. 

In table 3-17 the percentage standard error corresponding to the sixteenth 
test phase of the lambda set 6, NT. = 20 simulations is 24^. By equation 
3.48 the standard deviation of the instantaneous failure rate model's 
determined instantaneous failure rate for this phase is then 



S.D 




P.S.E. X X X 

'l6 16 



100 



24 X 0.24 
100 



0.0576 . 



(3.55) 



A quick examination of tables 3.15 thru 3-26 reveals that the 
goal of 30% percentage standard error of failure rate estimates utilized 
in evaluating the variability performance of the cumulative failure rate 
reliability growth model (chapter III. A. 5. b.) will be too restrictive for 
evaluating the instantaneous failure rate model's variability performance. 
Also, unlike the variability performance of the cumulative failure rate 
model which almost uniformly improved as test phases of the testing pro- 
cedure were completed, variability performance of the instantaneous model 
oscillates for many lambda sets. For example, keeping In mind that model 
variability performance begins with the percentage standard errors re- 
corded for phase 2, the instantaneous model's variability performance for 
lambda set 1, five tests per phase, determined failure rate in table 3.15 
improves steadily from 76% in phase 2 to 38% in phase 11. In phase 12 
the model's variability performance jumps up to 41%, improves back to 
32% by phase 14, and finally, climbs to 35^ for the last test phase, 
phase 16. 



141 



This oscillation in the variability performance of the in- 
stantaneous failure rate model means that even if a reasonable percent- 
age standard error benchmark is selected, determining the first phase 
for which variability performance on all lambda sets is within the goal 
does not guarantee the performance goal will be satisfied for all subse- 
quent test phases of the acquisition cycle. 

Tables 3-15, 3-16, and 3-17 display the instantaneous failure 
rate reliability growth model's variability performance for determined 
instantaneous failure rate estimates for the sixteen test phase proce- 
dures. If a percentage standard error goal of k0% is set, the goal is 
never achieved for NT. = 5 or NT. = 10 (tables 3-15 and 3- 16), but is 
achieved for NT. = 20 (table 3-17) on test phases 5, 7, 8, 14, and 16. 
Again, the oscillating character of the instantaneous model's variability 
performance is displayed by this method of examination. Tables 3-18, 
3 . 19 , and 3.20 show that in the case of the contracted six test phase 
procedures the instantaneous failure rate model's variability performance 
for determined instantaneous failure rate fails to attain the k0% goal 
for all simulated tests per phase specifications; i.e., NT. = 5, lO, and 
20 . 

In both the sixteen and six test phase reliability testing 
procedures determined failure rate estimates given by the instantaneous 
model were least accurate for lambda set 8 and M0D8. See figures 3-^3, 
3 . 68 , and 3-69. Also, the model's accuracy performance for these lambda 
sets was the only case in which no definite bias trend was evident. The 
instantaneous failure rate model's variability performance for determined 
instantaneous failure rate estimates is almost uniformly the worst for 
lambda sets 8 and M0D8 as evidenced in tables 3-15 thru 3-20. This was 
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to be expected based on the model's accuracy performance for these par- 
ticular specified underlying instantaneous failure rate paths. 

Tables 3-21 thru 3-26 contain variability performance results 
of the instantaneous model's forecast failure rates for both the sixteen 
and six test phase reliability testing procedures. Percentage standard 
error entries of 1000% indicate that the actual percentage standard 
errors in these cases were greater than or equal to 1000%. As with the 
cumulative failure rate model, lambda set 7, 12, 1^, M007, M0D12, and 
MODIA provide difficulty for the instantaneous model. These lambda sets 
are the ones which take the most rapid "plunge" to a low specified instan 
taneous failure rate of = 0.0500 or ^ ^ ~ 0.0500. Again, if these 

lambda sets are not considered, then a ^0% percentage standard error goal 
is achieved on phase lA for NT. = 5 and 10 and on phase 11 for NT. = 20 

I i 

in the case of the sixteen phase reliability testing procedure simula- 
tions. The instantaneous model's variability performance for forecast- 
ing during six test phase cycles is very poor achieving the A0% goal on 
the last test phase of the "most data" case (NT. = 20) for only seven of 
the fourteen lambda sets. 



C. DISCRETE RELIABILITY GROWTH MODEL 
1 . Model Description 

The discrete reliability growth model evaluated is of the form 



R 

m 



1 

exp (a + 3m) 



(3.56) 



where R = component reliability after the modification of the 
component 

m = number of modifications that have been made to the component 
and 

a, 6 = constants that determine the change in the model estimated 
component reliability as modifications to the component are 
accompl i shed . 
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The modification number m takes on values 0, 1, 2, . • . with m = 0 
designating the original version of the component. Component reliability 
may be thought of as an instantaneous or modification reliability as 
opposed to a cumulative or average reliability since it is the intrinsic 
reliability of the m^*^ modification version of the item rather than a 
character i St i c reliability for all m+1 versions of the item that have 
been produced. Under the assumption of reliability growth, the quantity 
exp (a + 3m) in equation 3-56 is expected to increase as modifications 
to the component are made which hopefully improve its reliability. Hence, 
as modifications are accomplished the fraction 1/exp (a + 3m) in equation 
3.56 decreases and component reliability increases toward the maximum 
reliability of 1.0. The model is considered discrete since the single 
independent or control variable m, modification number, is discrete as 
opposed to a continuous variable such as total accumulated test time. 

A mathematical analysis of the discrete reliability growth model 
is given in reference 1. Other Monte Carlo simulation evaluations and 
comparisons of the discrete model are contained in references 5 and 9 . 

2 • Reliability Testing Procedure 

For the discrete reliability growth model the appropriate reli- 
ability testing procedure of a system acquisition cycle is one in which 
modifications to an item are made as a specified number of failures of 
the item are observed during the testing procedure. The modifications 
of the item are made to improve reliability and hopefully result in such 
improvement. Modifications are indexed by m with a total of NTM modi- 
fications being considered during an acquisition cycle; i.e., m = 0, 1, 

2, 3, ••• , NTM. The total number of item failures that are observed 

th 

during testing of the m modification version of an item before the item 
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\im: 





is modified again is denoted by NTF . The total number of item fail- 

ures specified before reliability testing is conducted and may 

be varied from modification to modification. During testing of each 

modification version of an item, an underlying, inherent reliability 

that is unknown to the project managers/contractors is present in the 

item. The underlying modification reliability of the m^^ modification 

version of the component is denoted as . 

t h 

Between each failure of the m modification version of an item 

a number of tests of the item are conducted which are denoted as NT .; 

m,j 

i.e., the number of tests run on the m^*^ modification version of the 

th th 

item between the (j-l) failure and the j failure. NT . includes 

m,j 

th 

the j failure. So, NT , is the number of tests performed thru the 

m , 1 

first observed failure of the item when the underlying modification reli- 
ability i s R . NT - is the number of tests performed from the first 
m m,2 ^ 

failure thru the second failure of the item while the underlying modifi- 
cation reliability remains R and so on. After NTF failures of the 

m m 

item under test are observed, a modification is accomplished and the 

underlying modification reliability of the component changes to • 

Therefore, the failure number index j runs from 1 to NTF for each 

m 



modification m ; i.e., NT , , NT NT . All NT . are 

m , I m , z m , NTF m , j 

m 

random variables and cannot be specified beforehand. This is quite dif- 
ferent from the testing procedure for the continuous reliability growth 
models evaluated where the number of tests per test phase were specified 
prior to any testing being performed. 

Figure 3-78 is a schematic diagram of the discrete model reli- 
ability testing procedure. The total number of modifications NTM and 

the total number of failures permitted before modification NTF are 

m 
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DISCRETE RELIABILITY GROWTH MODEL RELIABILITY TESTING PROCEDURE DIAGRAM 



specified prior to testing. The number of tests performed between fail- 
ures NT . for each modification version of the item under test consti- 
m,j 

tute the data collected from the reliability testing procedure. As with 

the continuous reliability growth models the most salient feature of the 

computer simulation of the discrete model reliability testing procedure 

is that the analyst also specifies the underlying modification reliability 

R that is effective for each modification version of the item. Thus an 
m 

accurate assessment of the accuracy of the model can be made because the 

R values are known, 
m 

3 . Reliability Testing Procedure Computer .Simulation 
a. Summary 

The reliability testing procedure appropriate to the discrete 
reliability growth model was simulated on the Naval Postgraduate School 
VV. R. Church Computer Center's IBM 360/67 System utilizing standard com- 
puter simulation techniques. Reliability testing procedure design para- 
meters NTM and were specified along with various underlying modi- 

fication reliability sets. Tests of components were then simulated with 
component success or failure being determined for each test based on the 
appropriate underlying modification reliability R^ . Number of tests 

between failures NT . data were collected as each procedure simulation 

m,j 

was accomplished. Simulations of reliability testing procedures for each 
specified modification reliability set were replicated one-hundred times 
in order to assess the discrete reliability growth model's variability 
performance. 

After the reliability testing procedure simulations were com- 
pleted, discrete reliability growth model estimates of the underlying modi 
fication reliability of the component were produced based on the simulated 
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test data by computations utilizing ordinary least squares regression 
techniques to estimate the model parameters a and B • Mean and 
variability statistics of the model reliability estimates were then com- 
puted to permit performance evaluation of the discrete reliability 
growth model . 

b. Detail Description 

The computer simulation of the reliability testing procedure 
appropriate to the discrete reliability growth model was initiated by 
reading in the following reliability testing procedure design specifi- 
cations; NTM, the total number of modifications to be made for the 

acquisition cycle under consideration; MTF^, the total number of failures 
th 

of the m modification version of the item to be observed until the 
th 

(m+1) modification is made; the underlying, intrinsic modification 

th 

reliability of the m modification version of the item; and NS I MS, the 
number of times the specified reliability testing procedure was to be 
simulated. For this thesis NSIMS was always specified as one-hundred 
s imu 1 at i ons . 

After specification data were read into the computer, uniform 
(0,l) random variates were drawn in sequence corresponding to component 
tests starting for the modification 0 type component; i.e., uniform 
random variates were drawn in sequence and indexed as U . . for m = 0, 

m , j , I 

1, 2, ..., NTM; j = 1, 2, 3, ••• , ~ '** starting 

with the , , test. 

Each uniform variate was then compared with the appropriate 

underlying modification reliability to determine if a success or a 

failure occurred during the test. The number of tests NT . and the 

m,j 

number of failures accumulated NF data were recorded after each simu- 

m 

lated test such that 
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(3.57) 



NT . = NT . + 1 (NT . incremented by one test), and 
m,j m,j m,j 



NF 



m 



NF i f U . . < R (success) 

m m , j , I m 



NF + 1 if U . . > R (failure) 
m m 9 j 9 I m 



(3.58) 



for m = 0, 1,2, ... , NTM; j = 1, 2, 3, ... . i = 1> 2, 3> 

... If the test was a success, then the simulation proceeded to the next 

test. If the test was a failure, then the number of failures accumulated 

NF^ was compared to the total number of failures until modification 

NTF to determine if a modification was appropriate. If NF was less 
m ^ m 

than NTF , testing continued for the given modification and number of 
m ^ 

tests were then counted until the next failure; i.e., NT . If NF 

m , J + I m 

was equal to NTF , then a modification of the item under test was simu- 
^ m 

lated and testing continued for the next modification; i.e., ^^d 

NT , being tallied next. 

Testing in this sequence was simulated until equal 

to NTF|^j.|.j^ which signalled the termination of a given reliability test- 
ing procedure computer simulation. The simulation resulted in the number 

of tests until a failure NT . data being recorded for m = 0, 1,2, ... , 

m,j 

NTM and j = 1, 2, 3, ... , NTF . These data constituted the total data 
gathered from a single computer simulation of the discrete model reli- 
ability testing procedure for a single specified underlying modification 
reliability progress path; i.e., a specified set of reliabilities which 
shall henceforth be referred to as a reliability set. The reliability 
testing procedure was then replicated one-hundred times (r = 1, 2, 3, ... > 
NSIMS = loo) such that reliability testing procedure results data were 

actually indexed as NT . . Since ten reliability sets were utilized 

m , J , r 
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for evaluating the discrete reliability growth model with three different 

total number of failures until modification NTF specified for each 

m 

reliability set, 3000 reliability testing procedures were simulated for 
the discrete reliability growth model evaluation. 

4 . Computer Simulation Data Manipulation 

The discrete reliability growth model of equation 3-56 is proposed 
as a model of the unknown, underlying modification reliability in- 

trinsic to a component as it proceeds thru a" system acquisition cycle. 

It is against this true reliability R^ that the model's performance was 
measured . 

Equation 3.56 may be given in the form 

R = 1 - exp (-a - 6m) or (3-59) 

m 

exp (-a - 6m) = 1 - R (3.6o) 

m 

for m = 0, 1, 2, ... , NTH. The logarithmic transformation of equation 
3.60 yields 



(a + 6m) . = - 1 n ( 1 - R ) . 

J m j 



(3.61) 



for m = 0, 1, 2, ... , NTM and j = 1, 2, 3, ... , NTF^ . 

An unbiased estimator of the quantity a + 6m is given in 
reference 1 as 



(a + 6m) . = 

m,j 



0.0 i f NT . = 1 (first test was failure) 
m, J 



(3.62) 



1 



l+2 + 3+ *"+fjT .-1 

m,j 



if NT . = 2 
m,j 



for m = 0, 1 , 2, . . . , NTM and j = 1 , 2 , 3 , • . . , NTF . 

m 
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By equation 3-61 



-ln(l-R) . = (a+ Bm) 

m m,j m,j 



for m=0, 1, 2, ... , NTM and j=l,2,3, ••• , NTF 
relation an average estimate may be computed as 



m 



NTF 



m 



ln(l-R ) = (a + Bm) = ^ (a + Bm) 

mm m N r * * 



m 



m 



j=i 



Letting 



Y = -1n(1-R ) , 

m mm 



then equation 3-61 may be written as 



Y = a + B m 
mm m 



for m=0, 1, 2, ... , NTM which is of the form Y = a 

ordinary least squares regression estimates as given in 

a and B estimates are 
m m 




^ (i - m) Y. 

i=Q 

m 

Z) 

1=0 



and 





m 



(3.63) 
From this 

(3.6A) 



(3.65) 

+ BX. Applying 
reference 5, the 



( 3 . 66 ) 



(3.67) 
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for m=l,2, 3, ••• , NTM where 



Y 



m 




and 



tn 



1 

m+1 



E 



i=0 



( 3 . 68 ) 



(3.69) 



Finally, these and ^ estimates are utilized in the discrete re- 

liability model, equation 3.56, to produce the model estimates of the 
unknown, underlying modification reliability. These estimates are given 
by 



R 



m 



1 



exp ( 'o' + ^ nn) 

m m 



(3.70) 



for m = 1, 2, 3. ... > NTM. Note than since the regression procedure re- 
quires a minimum of two observations, model reliability estimates are pro- 
duced from the first modification thru the last modification (NTM). A 
reliability estimate for original version of the component is derived 
from equation 3.56 and 3*64 as 




exp (a + Btn)^ 



(3.71) 



Again, since the reliability testing procedure was simulated one- 
hundred times (r = 1, 2, 3, ... » NSIMS = lOO) for each reliability set, 
one-hundred estimates of the modification reliability were obtained for 
each modification version of the component under test. The mean value, 
standard deviation, and percentage standard error of these estimates 
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were computed as 



_ NSIMS 

^m NS I MS ^ ^ *^m, r 

r=l 



(3.72) 



NSIMS 



S.D; 




{r 

m, r m 



and 



(3.73) 



P . S . E 



S D 

b.U.p, 

m 




X 100 



(3.7M 



form = 0, 1, 2, ... , NTM . 

To examine the discrete reliability growth model's forecasting 
capability a next modification reliability forecast was made for modifi- 
cations m = 2, 3, ••• > NTM. Forecasting started for the second 

modification since model parameter estimates were not available until 
testing under the first modification was completed. No forecast was made 
past modification NTM because there would be no underlying modification 
reliability with which to compare. Forecasts were made as 




exp ('a + 's' (m+1)) 

m m 



(3.75) 



for m = 2, 3, ••• > NTM. Finally, forecast modification reliability 

statistics similar to the determined modification reliability statistics 
in equations 3*72, 3*73, and 3-7^ were computed as 

NSIMS 

= Nsk E 

r=l 
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NSIMS 



S.O. 



FR /NSIMS-1 
m- 



V (FR - )^ , 

X-j m, r m ’ 



and 



r=l 



(3.77) 



S.D.' 



P . S . E . 



FR 



m 



FR 



m 



FR 



m 



X 100 



(3.78) 



for m=2, 3, ... , NTM. 

To evaluate the discrete reliability growth model ten reliability 
sets were formed for use as the specified underlying reliability progress 
paths. These reliability sets are listed in table 3-27. All ten reli- 
ability progress paths start at a relatively low reliability of 0.200 
for the original version of the component under scrutiny; i.e., R^ = 0.200 
for reliability sets 1 thru 10. The paths display reliability progress 
that ranges from extremely rapid reliability growth (reliability set 1; 

Rq = 0.200, R| = 0 . 925 ) to linear growth (reliability set 6; ~ ^m 

0 . 150 ) to permanently stagnated reliability progress (reliability set 10; 
Rq thru R^ = 0 . 200 ). Reliability progress was simulated only for compo- 
nents that were modified a total of five times during the acquisition 
cycle; i.e., NTM = 5 and m = 0, 1, 2, 3, 5. Each reliability set 

progress path was simulated for three different total number of failures 
until modification specifications; i.e., NTF^ = 1, 3, and 5 failures 
which were held constant from modification to modification. 

5 . Model Performance 

a. Accuracy Performance 

Figures 3*79 thru 3.98 present all the cases (ten reliability 
sets) of the discrete reliability growth model's capability to determine 
and forecast the unknown, underlying modification reliability progress 
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RELIABILITY SETS 
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path during a reliability testing procedure for total number of failures 
until modification NTF^ = 1 and S- These graphs display the "least data" 
and "most data" cases which can be contrasted for change and improvement 
as the testing sample is increased. For the NTF^ = 3 cases, the accu- 
racy performance fell uniformly between the MTF^ = 1 and NTF^ = 5 accu- 
racy performance. 

The graphs depict the specified true underlying modification 
reliability progress path ( , solid line), the mean model deter- 
mined modification reliability from equation 3-72 for each modifica- 

m 

tion version of the component under test (0 , circles), and the mean 
model forecast modification reliability from equation 3-76 for the 

second thru the final modification version of the component (X , crosses) 
plotted versus the modification number m = 0, 1, 2, 3> 5- Note that 

the point plotted for the mean model determined modification reliability 
of the original version (m = O) of the component under test is not model 
determined; rather, it is the mean value of the estimate given by the 
estimator of equation 3-71 • This point allows the accuracy of the reli- 
ability estimator utilized to be examined also. 

To illustrate those quantities graphed in the accuracy per- 
formance figures, observe on figure 3-83 that for the third modification, 
the specified underlying reliability was 0.90, the mean model deter- 
mined modification reliability was approximately 0.86, and the mean 

model forecast modification reliability FR^ made from the second modi- 
fication was approximately 0.11. Since this graph is for reliability 
set 3, referring to table 1>.T1, the specified modification reliability 
for the third modification of reliability set 3 is 0.900 as is graphed 
on figure 3-83- Finally, note on figure 3-83 for the unmodified version 
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of the component tested (m = O) that the reliability estimator of equation 
3 . 71 . produced a mean estimate of approximately 0.175 for the true 

underlying reliability of 0.200. 

With the exception of reliability set 9 (figures 3-95 and 
3 . 96 ) the discrete model determines the shape of the underlying reliability 
progress path with very good accuracy. Forecasting accuracy, while fairly 
good for the "nice" reliability progress path sets 1 thru h, displays 
difficulty with the anomalous reliability progress paths characterized 
in reliability sets 6 thru 10. The discrete model occasionally produced 
negative mean forecast estimates of reliability. The negative mean fore- 
cast reliability estimates, the magnitude of which never exceeded 0.20, 
are plotted as 0.0 on the accuracy performance graphs. For example see 
figure 3-97 for mean forecasts at modifications 2, 3, and 4. No negative 
mean forecasts of reliability were produced when the total number of 
failures until modification was specified as five “ 5); i.e., the 

"most data" case. 

The discrete model accuracy performance graphs display the 
marked improvement that takes place both in determining and forecasting 
reliability status as the specified total number of failures until modi- 
fication NTF is increased from one to five which essentially increases 
the test data which the model can utilize. The improvement in accuracy 
is also displayed in the NTF^ = 3 accuracy performance graphs which are 
not presented. The reliability estimator of equation 3-71 displays the 
same improvement in accuracy when the testing sample size is increased 
as may be seen by examining the graph pairs for MTF^ = 1 and MTF^ = 5 
at the modification m = 0 point. 

Figures 3-79 thru 3-98 reveal that the discrete reliability 
model's accuracy performance for determining the underlying reliability 
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is in general slightly pessimistic in the instances when mean estimates 
are not "on the money" as in figures 3-84 and 3-86. Except for the re- 
liability sets 1 and 2, = 5 cases shown in figures 3«80 and 3 - 82 , 

mean model forecasting accuracy tends to run from slightly pessimistic 
as in figure 3-90; reliability set 6, = 5 to grossly pessimistic as 

in figure 3 . 91 ; reliability set 7, ~ ^ ^^'Qure 3-97; reliability 

set 10, NTF = 1. The general trend evidenced by the graphs is that 
m 

the less test data available (NTF^ = l) the more pessimistic the mean 
model estimates, determined or forecast. The same is true of the reli- 
ability estimator of equation 3-71. 

Reliability set 9 which characterizes a reliability growth 
pattern interrupted by a period of reliability degradation is the only 
set with which the discrete model experiences a significant problem. 

While not detecting the full magnitude of the degradation, the model's 
mean determined reliability estimates do display the period of degrada- 
tion credibly. See figures 3-95 and 3-96. However, the model's mean 
forecast modification reliability estimates are quite contrary to the 
true underlying reliability progress path. The significant problem 
being that the period of degradation is not forecast until the true under- 
lying reliability status has recovered from the period of degradation and 
is in fact displaying significant reliability growth. This behavior is 
apparent even for the "most data" case NTF^ = 5 in figure 3>96. While 
model forecasting capability suffers in the "least data" case for reli- 
ability set 10, NTF^ = 1, the discrete model copes with permanent reli- 
ability stagnation satisfactorily as seen in figures 3-97 and 3 * 98 . 
b. Variability (Precision) Performance 

Tables 3«28 thru 3-33 present the discrete reliability growth 
model's variability performance for determining and forecasting 
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FIGURE 3.79 

DISCRETE RELIRBILITT GROWTH MODEL PERFORMANCE 
RELIABILITT SET 1: 5 MODS, 1 FflILURE/MOD 



171 



SPECIFIED/MERN RELIABILITY 

0.20 0.30 O.MO 0.50 0.60 0.70 0.80 0.90 



O 

o 




MODIFICRTION NUMBER 



FIGURE 3.80 

DISCRETE RELIABILITY GROWTH MODEL PERFORMANCE 
RELIABILITY SET 1; 5 MOOS, 5 FAILURES/MOO 
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FIGURE 3.81 

DISCRETE RELIRBILITT GROWTH MODEL PERFORMANCE 
RELIflBILITT SET 2: 5 MODS. I FRILURE/MOO 
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FIGURE 3.82 

DISCRETE RELIABILITY GROWTH HCOEL PERFORMANCE 
RELIABILITY SET 2: 5 MODS, 5 FAILURES/MOD 
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FIGURE 3.83 

DISCRETE RELIRBILITT GROWTH MODEL PERFORMfiNCE 
RELIRBILITT SET 3: 5 MODS, I FfliLURE/MOD 
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FIGURE 3.84 

DISCRETE RELIRBILITT GROWTH MODEL PERFORMANCE 
RELIABILITT SET 3; 5 MODS, 5 FP, I LURES/MOD 
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FIGURE 3.85 

DISCRETE RELIABILITY GROWTH MODEL PERFORMANCE 
RELIABILITY SET 4; 5 MODS, 1 FAILURE/MOD 
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SPECIFIED/MEflN RELIflBILITT 
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FIGURE 3.86 

DISCRETE RELIflBILITT GROWTH HODEL PERFORMANCE 
RELIflBILITT SET 4: 5 MOOS, 5 FflILURES/MOO 
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SPECIFIEO/MEflN RELIABILITY 

.00 0.10 0.20 0.30 0.«0 0.50 0.60 0.70 0.80 0.90 



o 

o 




. t SPECIFIED RELIflBILITT 
O I MEAN MODEL DETERMINED RELIflBILITT 
X 1 MEAN MODEL FORECAST RELIABILITY 



'^’.00 



1^00 



2 '. 00 3 '. 00 li'.OO 5 '. 00 

MODIFICATION NUMBER 



"7.00 



6.00 



FIGURE 3.87 

DISCRETE RELIABILITY GROWTH MODEL PERFORMANCE 
RELIABILITY SET 5; 5 MODS. 1 FAILURE/MOD 
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FIGURE 3.88 

DISCRETE RELIRBILITT GROWTH HOOEL PERFORMANCE 
RELIABILITY SET 5: 5 MODS, 5 FAILURES/MOD 
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SPECIFIED/MERN RELIABILITY 
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FIGURE 3.89 

DISCRETE RELIABILITY GROWTH MODEL PERFORMANCE 
RELIABILITY SET 6: 5 MODS. 1 FAILURE/MOD 
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FIGURE 3.90 

DISCRETE RELIABILITY GROWTH MODEL PERFORMANCE 
RELIABILITY SET 6: 5 MOOS, 5 FAILURES/MOD 
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FIGURE 3.91 

DISCRETE RELIflBILITT GROWTH MODEL PERFORMANCE 
reliability set 7; 5 MODS, 1 FAILURE/MOD 
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SPECIFIED/MEflN RELIABILITY 
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FIGURE 3.92 

DISCRETE RELIABILITY GROWTH MODEL PERFORMANCE 
RELIABILITY SET 7; 5 MOOS, 5 FAILURES/MOO 
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SPECIFIEO/MEflN RELIABILITY 
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FIGURE 3.93 

DISCRETE RELIBBILITY GROWTH MODEL PERFORMANCE 
RELIABILITY SET 8: 5 MODS. 1 FRILURE/MOD 
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FIGURE 3.94 

DISCRETE RELIABILITY GROWTH MODEL PERFORMANCE 
RELIABILITY SET 8: 5 MOOS, 5 FAILURES/MOD 
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FIGURE 3.95 

DISCRETE RELIABILITY GROWTH MODEL PERFORMANCE 
RELIABILITY SET 9: 5 MOOS, 1 FAILURE/MOO 
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SPECIFIED/MEflN RELIflBILITT 
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FIGURE 3.96 

DISCRETE RELIflBILITT GROWTH MODEL PERFORMANCE 
RELIflBILITT SET 9; 5 MOOS, 5 Ffl I LURES/MOO 
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SPECIFlEO/MEflN RELIABILITY 
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FIGURE 3.97 

DISCRETE RELIABILITY GROWTH HODEL PERFORMANCE 
reliability set 10: 5 MODS, 1 FAILURE/MOD 
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SPECIFIEO/MEfiN RELIABILITY 
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FIGURE 3.98 

DISCRETE RELIABILITY GROWTH MODEL PERFORMANCE 
RELIABILITY SET 10; 5 MOOS. 5 FAILURES/MQD 
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the reliability status of a component as it undergoes modification in a 
system acquisition cycle. Entries in the tables are percentage standard 
errors as computed in equations 3*7^ and 3*78. For example from figure 
3.83 the mean model determined reliability for modification 3 from reli- 
ability set 3 was found to be =0-86 in the NTF^ = 1 case. From 
table 3.28 the percentage standard error P.S.E.'^ corresponding to this 

''3 

mean model determined modification reliability is 20Z. Therefore, by 
equation 3.7^ the standard deviation for the observed mean model deter- 
mined reliability is 

^-5 on 

^ = = . (3.79) 

In. tables 3-28, 3-29, and 3-30 the discrete model 's vari- 
ability performance for determining modification reliability status is 
displayed for the specified total number of failures until modification 
cases of one, three, and five failures, respectively. Two salient 
characteristics displayed by the results are (l) the large variability 
of the estimates produced by the reliability estimator of equation 3-71 
and ( 2 ) the large variability of the reliability estimates produced by 
the discrete model for reliability set 10, the permanently stagnated 
reliability case. The same difficulty is displayed to a lesser degree 
for reliability set 7- 

If reliability set 10 is not considered, then a goal of k0% 

percentage standard error is achieved by the discrete reliability growth 

model on the fifth modification for the NTF = 1 and 3 cases and on the 

m 

fourth modification for the NTF = 5 cases. If reliability set 7 is 

m ' 

also dropped from consideration, the h0% goal is achieved on the fifth 



191 



modification for NTF = 1 , on the third modification for NTF = 3, and 

m m ’ 

on the second modification for NTF = 5- Note that the reliability 

m 

sets characterizing temporary reliability stagnation and degradation 

(sets 8 and 9) did not have to be taken out of consideration in order to 

find variability performance at a reasonable level. 

Tables 3*31 » 3*32, and 3-33 present the discrete model's 

variability performance in forecasting the reliability of the subsequent 

modification version of the component under test for the NTF =1,3, and 

m 

5 cases. Again, the worst variability performance occurs for reliability 
sets 7 and 10. Dropping reliability sets 7 and 10 from consideration, a 
k0% percentage standard error goal is achieved only in the case of 
NTF^ = 5 at the fourth modification. Hence, variability performance of 
the discrete reliability growth model for forecasting is not particularly 
good. 
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IV. CONTIMUOUS FAILURE RATE RELIABILITY 
GROWTH MODELS COMPARISON 



Codier in reference 2 gives a derivation in which an equation of the 
form of the continuous instantaneous failure rate reliability growth 
model (equation 3.37) is obtained by differentiating the equation of the 
continuous cumulative failure rate reliability growth model (equation 
3.1) with respect to the total accumulated test time TT after substitu- 
tion of the cumulative failure rate estimator of equation 3.13. The 
derivation essentially proceeds along the following line: 



_ TF 
TT " TT 


BTT ^ ; therefore, 


(4.1) 


TF = BTT^’ 


1 -a) 


(4.2) 



Now, 



3TF 

3TT 



8 

3TT 



(6TT^’~“^) = (l-a)BTT''^ 



and 



(^.3) 



X 



T 



= 9TF 
■ 3TT * 



Hence Codier derives an equation 






= (l-a)3TT'“ {k.5) 

and claims that the equation is the continuous instantaneous failure 
rate model 



X^ = ( 1-a) b TT ^ ; i .e, , 



(4.6) 
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equation 3-37. The implication being that the cumulative model parameters 
a and 3 sre equivalent to the instantaneous model parameters a and 
b , respectively; and so, once either a and 3 or a and b have 
been estimated, it is unnecessary to estimate the other parameter pair. 

The cumulative and instantaneous failure rate models were considered 
as two unrelated models which were proposed to model different aspects of 
reliability growth from the viewpoint of this thesis. The evaluation of 
the two models was conducted accordingly. In order to investigate the 
proposition that the cumulative and instantaneous models are related thru 
equivalence of model parameters a hybrid cumulative failure rate model was 
devised that utilized the parameter estimates generated for the instanta- 
neous failure rate model. Also, a hybrid instantaneous failure rate 
model was devised that utilized the parameter estimates generated for 
the cumulative failure rate model. Specifically, the hybrid cumulative 
failure rate model is given by 




where a. and d. were determined by the instantaneous failure rate 
model equations 3-^2 and 3-^3- The hybrid instantaneous failure rate 
model is given by 






r 



TT. 







(4.8) 



where and "S'. were determined by the cumulative failure rate 

I ,r ,r 

model equations 3.21 and 3*22 after appropriate transformations given 
i n chapter I I I .A. 4 . 
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As cumulative and instantaneous failure rate model parameter esti- 
mates were determined for each phase of each simulation for the differ- 
ent lambda sets they were applied to the hybrid models as well as to 
their appropriate models. Hybrid model estimates of the cumulative and 
instantaneous failure rates were recorded and the standard statistics 
of chapter III were compiled. 

Table 4.1 contains the mean cumulative and instantaneous determined 
failure rate estimates produced by the hybrid models for the lambda set 
3, ten tests per phase (NT. = lO) case listed with the appropriate under- 
lying mean test time weighted average cumulative failure rates Xjj 

i 

(equation 3.30) and the specified underlying instantaneous failure rates 
A. (table 3.1). Lambda set 3 is one of the "nice" failure rate sets 
for which both the cumulative and i ns tan taneous failure rate models dis- 
played some of their best accuracy and variability performance (see 
figures 3.5, 3.6, 3.7, and 3.44; and tables 3-4 and 3.16). Entries of 

10.0000 for the hybrid cumulative model mean determined cumulative failure 
rate in table 4.1 indicate that the actual mean estimate was greater than 
or equal to 10.0000. 

The performance of the hybrid models displayed in table 4.1 for a 
"nice" lambda set is typical of the behavior of the hybrid models for 
all the lambda sets, nice or anomalous. The hybrid cumulative model 
(instantaneous model's estimated parameters substituted into the cumula- 
tive model) produced failure rate estimates of magnitude greater than 

10.0 for the first few test phases followed by very erratic estimates 
(including negative mean estimates) which displayed no relation to the 
mean test time weighted average cumulative failure rates supposedly being 
modelled. On the other hand the hybrid instantaneous model (cumulative 
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TABLE A.l 



HYBRID 


CUMULATIVE AND 
LAMBDA 


INSTANTANEOUS FA 1 
SET 3: 16 PHASES 


LURE RATE MODEL 
, 10 TESTS/PHASE 


SIMULATION RESULTS 


PHASE 


MEAN TEST 
TIME 

WEIGHTED 

AVERAGE 

CUMULATIVE 

FAILURE 

RATE 

j- A y- y- 


HYBRID 

CUMULATIVE 

MODEL MEAN 

DETERMINED 

CUMULATIVE 

FAI LURE 

RATE 

y. y^ .l y^ y> y« 


SPECIFI ED 
INSTANTANEOUS 
FAI LURE 
RATE 

y^ y- y- y- y- y- ju y- y^ y- ju y- y- 


HYBRID 

INSTANTANEOUS 
MODEL MEAN 
DETERMINED 
1 NSTANTANEOUS 
FAI LURE 
RATE 

.u y- y- y- y- y- y- y- y* y- y- y- jL 


1 


0.7000 


0.6702 


0.7000 


0.6702 


2 


0.4703 


10.0000 


0.3530 


0.3605 


3 


0.3565 


10.0000 


0.2380 


0.3048 


k 


0.2863 


10.0000 


0.1800 


0.2669 


5 


0.2392 


10.0000 


0.1450 


0.2362 


6 


0.2064 


-0.1003 


0.1220 


0.2173 


7 


0.1818 


1.9321 


0.1060 


0.2016 


8 


0.1625 


-0.0171 


0.0933 


0.1889 


9 


0.1474 


-0.0065 


0.0837 


0.1796 


10 


0.1346 


0.0532 


0.0760 


0. 1684 


1 1 


0. 1241 


-0.0206 


0.0697 


0.1626 


12 


0.1152 


0.0237 


0.0644 


0.1548 


13 


0.1076 


0.0070 


0.0600 


0. 1497 


14 


0. 1009 


0.2491 


0.0562 


0.1442 


15 


0.0952 


0.0518 


0.0529 


0.1402 


16 


0.0900 


0.0060 


0.0500 


0.1357 
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model's estimated parameters substituted into the instantaneous model) 
produced what appeared to be a good failure rate estimate for the first 
one or two phases (i = 2, 3; models do not estimate for the first phase) 
but quickly transitioned to a dampened, sedate pattern of estimates which 
was very un response i ve to even the most radical variations in the under- 
lying instantaneous failure rate progress path* Also, the hybrid in- 
stantaneous model's failure rate estimates after the second or third 
phase generally exhibited increasing magnitude error except for crossing 
situations in the cases of temporary stagnation or degradation of the 
specified underlying instantaneous failure rates* 

Regarding the negative failure rate estimates produced by the hybrid 
cumulative model (instantaneous model parameters in cumulative model), 
the following observations on the parameter estimates and 

of the continuous instantaneous failure rate model were made during the 
reliability testing procedure computer simulations* During the simula- 
tions the magnitude of the parameter estimate ^ was continually very 

close to 1*0 often being more or less than 1*0 only in the tenth, 
eleventh, or twelfth decimal place. From the instantaneous model equa- 
tion 4*6 it can be seen that logically if the estimate is less 

I ,r 

than 1.0, the estimate should be positive; and, if the esti- 

mate is greater than 1*0, the ^ estimate should be negative in order 

to produce positive estimates of the i ns tan taneous failure rate* Exactly 
such behavior was observed during the computer simulations; as the 
magnitude of the ^ estimate moved to either side of 1*0 the sign 

of the ^ estimate would "flip-flop" accordingly. Of course if a 
negative ^ estimate is plugged into the cumulative failure rate model 

for 6 jn equation 4*1, a negative estimate of the cumulative failure 
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rate necessarily results no matter what the sign or magnitude of the 

a^ estimate which is substituted for a (save a value of^^ = <») • 

I , r i ,r 

Hence, it is very plausible for the hybrid cumulative model to have pro- 
duced negative estimates of cumulative failure rate. 

In contrast to the hybrid cumulative model's erratic failure rate 
estimates the hybrid instantaneous model (cumulative model parameters 
in instantaneous model) produced failure rate estimates which, after one 
or two test phases, displayed a "sluggish" growth pattern and generally 
increased in magnitude error from the underlying instantaneous failure 
rate X. being modelled. This situation is not surprising because the 
cumulative model parameters being employed in the hybrid instantaneous 
model are parameters utilized in modelling a "smoothed" quantity, the 
underlying cumulative (average) failure rate that is collectively charac 
teristic of all the versions of an item tested thru a given point in the 
acquisition cycle. As the history of the item stretches farther and 
farther (total accumulated test time TT increases) the cumulative 
failure rate of the item becomes increasingly insensitive to the current 
instantaneous failure rate intrinsic to the item. Hence, the parameters 
of the cumulative model likewise become increasingly insensitive to the 
current instantaneous reliability status. When these relatively stable 
parameters are utilized in another model whatever its form, "smoothed" 
stable output is reasonably expected. 
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V. EVALUATION SUMMARY AND CONCLUSIONS 



This thesis examined three reliability growth models for which accu- 
racy, precis ion, and robustness character i s t i cs were investigated over a 
wide variety of true underlying growth patterns, both “nice” and anomalous. 
The evaluation of a continuous cumulative failure rate model, an instanta- 
neous failure rate model, and discrete reliability model was accomplished 
by computer simulation of reliability testing procedures that were appro- 
priate to the reliability growth model types for a systems acquisition 
cycle. Performance of the models both in tracking and predicting true 
underlying reliability progress patterns was measured for mean accuracy 
and variability (precision) over progress patterns which included most 
situations that can be encountered, both good and bad. The models' per- 
formance character i St i cs are summarized in the following paragraphs, 

A. CONTINUOUS CUMULATIVE FAILURE RATE RELIABILITY GROWTH MODEL 

The continuous cumulative failure rate model examined generally dis- 
played very good to excellent accuracy performance in tracking both “nice" 
and anomalous true underlying cumulative failure rate patterns. The 
cumulative model did exhibit difficulty in determining and forecasting 
true underlying patterns that characterize rapid reliability growth 
(failure rate decay) in the latter phases of the systems acquisition 
cycle. The difficulty was evidenced by mean model determi ned/forecas t 
cumulative failure rate estimates that diverged from the true underlying 
values on the pessimistic side; thus, the cumulative model provided esti- 
mates that could be considered as upper bounds on failure rate in this 
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situation. The other anomalous situation with which the cumulative model 



experienced difficulty in tracking and predicting accuracy was one of 
reliability growth interrupted by a period of reliability degradation. 

The modelled cumulative failure rate path both lagged the true underlying 
cumulative failure rate path and failed to reflect the full magnitude of 
the failure rate degradation. Thus dependence on the model in this situ- 
ation could hamper response to the degradation by giving a “too little 
too late“ signal of the problem. The cumulative model coped with true 
underlying cumulative failure rate patterns that were permanently stag- 
nated quite adequately. 

As the reliability testing procedures of the acquisition cycles com- 
pleted more test phases, the cumulative model's variability (precision) 
performance improved un i formly for both “nice" and anomalous true under- 
lying failure rate paths. Variability percentage standard error goals 
were satisfied earlier and earlier in the acquisition cycle as more 
testing data was made available to the model. This good variability 
performance provides a degree of confidence in employing the cumulative 
failure rate model in actual systems acquisition programs where the 
intrinsic cumulative failure rate status is truly unknown. 

B. CONTINUOUS INSTANTANEOUS RAILURE RATE RELIABILITY GROWTH MODEL 

The continuous instantaneous failure rate model examined, while 
exhibiting the capability to track and predict the shape of various 
unknown, underlying instantaneous failure rate path types with very good 
accuracy and robustness, consistently produced optimistically biased 
estimates of the instantaneous failure rate. If bias must be accepted 
in a reliability growth model, then pessimistic bias is preferred to 
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optimistic bias. Consistent pessimistic bias permits managers to utilize 
the model produced estimates as lower bound type values and have confi- 
dence that reliability status is being observed from the “right side of 
the fence". On the other hand consistently optimistic model produced 
estimates impart a great deal of uncertainty as to how bad reliability 
status might be and what degree of corrective action is required. 

The anomalous situations in which the instantaneous model experienced 
difficulty in tracking and predicting the shape of the true failure rate 
path were (1) the initial period of recovery in reliability growth after 
a period of temporary reliability stagnation, (2) permanently stagnated 
reliability growth at high failure rates, and (3) reliability growth i n- 
terrupted by a temporary period of reliability degradation. In the case 
of temporary reliability degradation the instantaneous model charted a 
determined failure rate path that was an exaggeration of the underlying 
instantaneous failure rate path; i.e., the periods of growth were displayed 
optimistically and the period of degradation was portrayed to a magnitude 
greater than it was in truth. The instantaneous model's forecasts, while 
accurately capturing the shape of the underlying progress path, were con- 
sistently optimistic during the period of degradation. 

The instantaneous model's accuracy performance was generally better 
for contracted acquisition cycles and suffered when the reliability test- 
ing procedure was extended to encompass a large number of test phases. 
Because of the model's consistent optimistic bias, accuracy performance 
improved as the true underlying failure rate decreased thereby "sandwich- 
ing" the model estimates between the true failure rate and 0.0. Also, 
the instantaneous model routinely furnished "of f- the-sca 1 e" forecasts for 
the first one or two phases of an acquisition cycle. 

The instantaneous model's demonstrated capability to determine and 
forecast the true underlying reliability status was overshadowed by its 
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poor variability performance which (l) never achieved a really comfort- 
able percentage standard error goal uniformly for all failure rate pat- 
terns simulated and, more damaging, (2) oscillated as test phases of the 
reliability testing procedure were accomplished. Because of this poor 
variability performance, the instantaneous failure rate model cannot be 
employed by itself with any degree of confidence. 

C. DISCRETE RELIABILITY GROWTH MODEL 

The discrete reliability growth model examined generally displayed 
very good determined reliability accuracy performance even for the most 
restricted test data cases. Slight pessimistic bias was demonstrated for 
model determined reliability status while a greater magnitude of pessi- 
mistic bias was present in model forecast reliability. Again, pessimis- 
tic bias is favored over optimistic bias in charting a reliability prog- 
ress path. The degree of pessimism in determined and forecast reliability 
decreased markedly as more test data on each modification version of an 
item were gathered. In cases of limited test data the discrete model 
often forecast negative estimates of reliability for the early modifi- 
cation versions of a component under test. 

For the anomalous situation of temporary reliability degradation the 
discrete model (l) failed to determine the full magnitude of the degrada- 
tion and (2) provided reliability status predictions that lagged the 
true underlying reliability path outcomes significantly. In actual test- 
ing this performance characteristic would give a delayed signal of a 
situation that required corrective action^ Also, gathering additional 
test data failed to remedy this deficiency in the discrete model's per- 
formance. 
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Variability performance of the discrete model, while generally good, 
revealed difficulty with the permanently stagnated reliability case and 
the case of rapidly increasing re 1 i ab i 1 i ty during the latter modifica- 
tion versions of an item. Variability performance improved uniformly for 
all progress path character i zat i ons as more modifications of the item be- 
ing tested were accomplished. Also, variability goals were satisfied 
earlier in the acquisition cycle as more complete testing was accomplished 
on each modification version of the item. This nice behavior of the dis- 
crete model lends confidence to its utilization; and therefore, the dis- 
crete model is preferred to the continuous i ns tan taneous failure rate 
model for obtaining a measure of the current or "instantaneous” re 1 i - 
ability status of an item proceeding thru an acquisition cycle. 

D. GENERAL OBSERVATIONS 

For determining current reliability status it may be suggested that 
rather than employing the reliability growth models, why not utilize the 
point estimators of reliability status which were used to provide data 
to the models? Although the performance of the point estimators appro- 
priate to each reliability growth model were only observed at one point 
for each model in the reliability testing procedure simulations (first 
phase of testing or mod 0 version of a component), at that point the 
estimators displayed very poor variability performance upon which the 
reliability growth models improved rapidly and significantly which tends 
to negate any confidence in utilizing the pojnt estimators based on their 
good accuracy performance. 

In retrospect the performance character i s t i cs demonstrated by the 
reliability growth models examined suggest that as a model is employed 
the estimated reliability progress path it produces be compared with the 



209 



appropriate model accuracy performance graphs in chapter III, If the 
estimated reliability progress path corresponds to a case for which the 
computer simulation performance results show the model’s performance 
was good, then confidence can be placed in the estimated progress path. 

On the other hand if the estimated path corresponds to one of the 
anomalous cases where simulation results revealed a deficiency in the 
model’s performance, then the simulation performance results at least 
give an indication of which direction the true underlying reliability 
progress path lies. 

Finally, although use of the continuous i ns tan taneous failure rate 
reliability growth model is questionable, since the test data required 
for the i ns tantaneous model is identical to the data collected for the 
continuous cumulative failure rate model, application of the instanta- 
neous model simultaneously with the cumulative model utilizing the method 
described in the preceding paragraph may provide some insight to the 
shape of the true underlying i ns tantaneous failure progress path from 
the instantaneous model. But the comparison performed between the two 
continuous failure rate models definitely indicates that the cumulative 
failure rate and instantaneous failure rate models’ parameters not be 
interchanged based on their hypothetical equivalence. 
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APPENDIX A 



Derivation of Ordinary Least Squares Regression Estimates of the 
Continuous Instantaneous Failure Rate Reliability Growth Model Parameters 



The continuous instantaneous failure rate model reliability growth 
model as given in equation 3-37 may be written as 



A. = b(I-a)TT.~^ . 

I I 



(A.l) 



Taking the logarithmic transformation of this equation yields 



In A. = In b(l-a) - a In TT. 

I 1 



(A. 2) 



Let 



i = l 



Y. = In A. , 

I 1 



X. = In TT. , 

I I 



Y = 



E' 



, and 



X = 






Therefore, 



Y. = In b(l-a) - aX. 

I I 



(A. 3) 
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The residual error for regression is 



£. = Y. - Y. = Y. - [In b(l-a) - aX.] . {^.k) 



The goal of the regression is to minimize the sum of the residual errors 
squared; i . e. , 



m I n 






(A. 5) 

Taking partial derivatives of equation A. 5 with respect to b and a ; 
then equating the partials to 0 yields the following two equations in 
two unknowns: 

(1) a 

-^5 = ^(2 (Y, -In b(l-a) . aX.)(- 5 ^ (l-a)] -0 

(2) 3 

- 5 ^ [2 (Y; -In b(l-a) + aX,)(- j^(-b)+X,)] - 0 

Simplification of (l) and (2) yields in sequence: 



(r) 


Zi\- 


In b(l-a) + aX.] 


= 0 , and 


(2') 






1 


■In b ( 1 -a) + aX. ) 




d") 




y^l n b ( I -a) - 




(2") 










2^ mi 


y.+ Y' Y.X. = 

1 I 1 


Ziw 



In 



b(l-a) + ^X. In b(l-a) 



aX. 

12 jT^- 
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O'") 



y , = n In b(l-a) - a ^ ] X. , and 



(2'") XTZTT - 



^ I ^ y; - y In b(l-a) + (In b(l-a)) ^ 

(w E=<i - - E • 



Equation (1"') may be manipulated such that 

(1"") In b(l-a) = - y'v. + - Vx. = Y + aX or 

n ' I n Z—/ I 

(1""') b(l-a) = exp (Y + aX) . 

Finally, the estimate for b is 

exp (Y + X) , 

Substituting equation A. 6 into (2"') yields 



(A 



(2"") 



TTiT 






l^[exe_il_+^^ (1-a)] > X. 



(1-a) 



(1-a)] 



TT^ 



E^, 



Clearing the logarithms in (2"") and transposing leaves 



(2 ) (Y + aX) + (Y + aX) ^X. - 



Tl^ Ex. - ^ Ex. 



THTT 



Ex. - Ex. X. 



Expanding (2""') yields 



Ti^ X X . Y 2 X. + ax 2:x. - ^ X)Xl 



Ex.^-t^ Ex,= E 



Y. X. . 

I I 
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Substituting and clearing terms in (2*) leaves 



ttW E''! " tt^st E^i " 






E^' - TT^ E'', - E^Xi 



Solving for a in (2*'->) gives 



( 2 *,v,v) a(X ^X. - 2^X.^) = I]y.X. - ^ 



a = 



EXiXi - V ^x, 



Exi- E: 



(A. 7) 
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